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Abstract 

The  protest  process  is  a  means  of  ensuring  that  the  Government  conducts 
procurements  in  accordance  with  statutory  and  regulatory  procedures.  Additionally, 
protests  serve  a  vital  role  in  assuring  full  and  open  competition  in  the  federal  acquisition 
process.  Protests  can  be  costly  in  many  aspects,  including  the  direct  costs  of  taking  the 
action  as  well  as  more  indirect  costs  such  as  program  delays  and  damaged  business 
relationships.  The  Government  recognizes  the  negative  impact  protests  have  on  the 
procurement  process  and  has  enacted  several  laws  and  regulations  in  an  effort  to  reduce 
protest  frequency  and  streamline  procurement  procedures.  While  it  appears  that  the 
Government  has  made  substantial  efforts  aimed  at  reducing  protests,  it  becomes 
necessary  to  ask  whether  they  were  successful  in  their  efforts.  This  study  examined  one 
such  piece  of  legislation,  the  Federal  Acquisition  Streamlining  Act  of  1994  (FAS  A),  and 
foimd  that  there  was  an  impact  on  particular  contracting  practices.  Additionally,  the 
findings  of  this  study  indicate  that  protest  frequency  within  the  Air  Force  has  decreased 
since  FASA’s  implementation.  This  correlation  suggests  that  FASA  did  have  an  impact 


on  reducing  the  frequency  of  protests. 


THE  IMPACT  OF  THE  FEDERAL  ACQUISITION  STREAMLINING  ACT  OF 


1994  ON  POST-AWARD  PROTEST  FREQUENCY 


I.  Introduction 

It  is  not  unusual  for  contractors  who  are  interested  in  winning  Government 
contracts  to  invest  a  great  deal  of  time  and  money  in  preparing  a  proposal  in  response  to  a 
Government  Request  for  Proposal  (RFP).  After  expending  considerable  resources 
preparing  their  proposals,  unsuccessful  bidders  want  to  know  why  they  were  not  awarded 
the  contract.  They  may  feel  that  the  process  was  unfair,  that  there  must  have  been  a 
mistake,  or  they  may  want  to  know  where  their  proposals  were  weak  so  they  may  be 
more  competitive  in  future  procurements.  Failure  to  provide  answers  to  unsuccessful 
bidders’  questions  regarding  the  source  selection  may  result  in  the  filing  of  a  protest. 
Problem 

Protests  can  be  costly  in  many  aspects.  The  most  obvious  costs  involve  the  actual 
financial  resources  required  to  complete  the  protest  process.  Protests  can  be  lengthy,  and 
obviously  incur  greater  costs  as  time  passes.  Additionally,  contract  performance  may  be 
suspended  during  protests,  preventing  timely  delivery  and  satisfaction  of  Government 
requirements.  When  these  delays  involve  major  programs,  the  resultant  costs  could 
conceivably  amount  to  millions  of  dollars.  There  are  also  indirect  costs  of  protests,  such 
as  the  possibility  of  creating  a  new  precedence,  the  creation  of  public  relations  problems 
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for  the  Government,  and  damage  to  the  relationship  between  the  protesting  contractor  and 
the  Government.  All  of  these  costs  may  adversely  impact  the  level  of  competition  in 
Government  procurements,  either  directly  or  indirectly. 

The  Government  recognizes  the  negative  impact  protests  have  on  the  procurement 
process.  Consequently,  the  Government  has  enacted  several  laws  and  regulations  in  an 
effort  to  reduce  the  number  of  protests  while  maintaining  the  integrity  of  the  procurement 
system.  Some  of  the  more  significant  laws  include  the  Competition  in  Contracting  Act  of 
1984  (CICA),  the  Federal  Acquisition  Streamlining  Act  of  1994  (FAS A),  and  the  Federal 
Acquisition  Reform  Act  of  1996  (FARA).  Each  of  these  acts  have  either  added, 
removed,  or  modified  pre-existing  acquisition  policies  and  regulations,  and  each  has  had 
differing  effects  on  the  frequency  of  bid  protests. 

The  intent  of  CICA  was  to  “increase  competition  in  the  award  of  Government 
contracts”  and  to  “eliminate  the  imbalances  in  the  protest  system,”  [Walsh,  June  1987: 

53].  Previous  research  on  the  impact  of  CICA  on  the  number  of  protests  indicated  that 
there  was  a  10-15%  increase  in  protests  in  the  years  following  CICA’s  enactment  [Walsh, 
June  1987:  130]. 

The  Federal  Acquisition  Streamlining  Act  made  significant  changes  in  the 
following  areas  of  acquisition:  task  and  delivery  order  contracting,  advisory  and 
assistance  services  contracting,  source  selection  procedures  and  debriefings,  bid  protest 
procedures,  contractor  past  performance  and  other  evaluation  factors,  new  micro  purchase 
procedures,  commercial  item  procurement,  small  business  awards,  and  several  other 
areas.  While  President  Clinton  proclaimed  FASA  as  the  most  significant  piece  of 
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procurement  reform  legislation  since  the  passage  of  CICA  in  1984,  many  others  have 
criticized  FASA  for  not  going  far  enough  [Acquisition  Streamlining  Institute,  1997,  p. 

13].  Some  critics  maintain  that  more  should  have  been  done  in  other  areas,  such  as  those 
aimed  at  reducing  the  frequency  of  bid  protests. 

The  changes  imposed  by  FARA  can  be  seen  as  further  adjustments  that 
complement  the  reforms  made  by  FASA.  These  three  acts  are  among  the  more 
significant  efforts  of  the  United  States  Government  to  reform  acquisition  and  reduce  post¬ 
award  protests.  While  it  appears  that  the  Government  has  made  substantial  efforts  aimed 
at  reducing  protests,  it  becomes  necessary  to  ask  whether  they  were  successful  in  their 
efforts.  FARA  just  went  into  effect  on  1  January  1997,  so  it  would  be  premature  to  study 
its  effects  due  to  lack  of  data.  FASA  on  the  other  hand  has  been  in  effect  for  over  two 
years  at  the  time  of  this  writing  and  sufficient  data  should  be  available  to  conduct  a 
preliminary  analysis  of  any  effects  it  may  have  had  on  protest  frequency. 

Objective 

The  objective  of  this  research  is  to  examine  the  impact  of  The  Federal  Acquisition 
Streamlining  Act  of  1994  on  post-award  protest  frequency  within  the  Air  Force.  In 
examining  this  topic,  the  following  primary  research  question  was  developed: 

Primary:  Has  The  Federal  Acquisition  Streamlining  Act  of  1994  impacted  post-award 
protest  frequency  within  the  Air  Force,  and  if  so,  how? 

In  order  to  address  the  primary  research  question  posed  in  this  research,  it  is 
necessary  to  formulate  subsidiary  research  questions.  The  results  of  the  subsidiary 
questions  will  be  used  to  answer  the  primary  question. 
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Subsidiary: 

1)  Has  FAS  A  impacted  the  United  States  Air  Force’s  source  selection  processes 
in  a  manner  that  would  significantly  affect  post-award  protest  frequency? 

2)  Has  FASA  impacted  post-award  protest  frequency  within  the  United  States 
Air  Force? 

Organization  of  the  Study 

To  accomplish  the  goals  of  this  study,  it  was  first  necessary  to  develop  an 
understanding  of  the  protest  process  and  how  protest  frequency  could  be  influenced  by 
events  occurring  during  the  source  selection.  Chapter  II  provides  this  background  and 
also  briefly  discusses  relevant  existing  statutes  and  the  evolution  of  Acquisition  Reform 
within  the  Federal  Government.  The  Literature  Review  conducted  for  chapter  II  revealed 
that  certain  provisions  of  recent  acquisition  legislation  were  conducive  to  reducing  bid 
protests  while  others  might  actually  incite  increases  in  the  same. 

Chapter  III  details  the  study  methodology.  It  provides  support  for  the 
appropriateness  of  a  case  study  methodology  and  statistical  analysis  of  archival  protest 
data  to  accomplish  the  goals  of  this  study.  Additionally,  chapter  III  discusses  the  details 
on  the  structured  interview  that  was  employed  and  provides  the  guidelines  that  were  used 
in  deducing  meaning  from  the  responses. 

Chapter  IV  summarizes  and  analyzes  the  interview  responses  and  results  from  the 
statistical  tests.  Chapter  V  is  the  final  chapter  and  discusses  the  conclusions  made  from 
the  data  collected.  The  conclusions  drawn  from  this  research  indicate  that  particular 
practices  such  as  the  content  and  administration  of  post-award  debriefs  has  been  changed 
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as  a  result  of  FASA’s  implementation.  As  a  consequence  of  these  changes,  the  number 
of  protests  have  decreased.  Surprisingly  however,  the  increase  in  opeimess  and  more 
thorough  debriefs  have  apparently  not  influenced  any  changes  in  the  quality  of  proposals 
submitted  by  potential  contractors.  The  study  concludes  with  recommendations  for  future 
research. 
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II.  Background 


Introduction 

There  are  currently  two  methods  available  to  the  Federal  Government  for  the 
competitive  procurement  of  goods  and  services:  sealed  bidding  and  competitive 
proposals  (negotiated  procurement).  In  determining  which  method  to  use,  “agencies  are 
required  to  use  the  competitive  procedure  or  combination  of  competitive  procedures  that 
is  best  suited  under  the  circumstances  of  the  procurement,”  (41  USC  §253,  and  10  USC  § 
2304).  These  statutes  also  state  the  criteria  for  using  sealed  bidding  and  requires  its  use  if 
the  following  criteria  are  all  present  [Cibinic  and  Nash,  1986:  290]: 

(2)  In  determining  the  competitive  procedures  appropriate  under  the 
circumstances,  an  executive  agency  — 

(A)  shall  solicit  sealed  bids  if— 

(i)  time  permits  the  solicitation,  submission,  and  evaluation  of  sealed  bids; 

(ii)  the  award  will  be  made  on  the  basis  of price  and  other  price-related  factors; 

(iii)  it  is  not  necessary  to  conduct  discussions  with  the  responding  sources  about 
their  bids;  and 

(iv)  there  is  a  reasonable  expectation  of  receiving  more  than  one  sealed  bid;  and 

(B)  shall  request  competitive  proposals  if  sealed  bids  are  not  appropriate  under 
clause  (A). 

Under  sealed  bidding  procedures,  the  contract  is  awarded  to  the  lowest  priced 
bidder  that  is  responsive  to  the  requirements  in  the  solicitation  and  is  determined  to  be 
responsible.  Bidders  may  witness  bid  openings  and  inspect  the  bids  and  accompanying 
documents  afterwards.  Thus,  unsuccessful  bidders  generally  know  they  were  not 
awarded  the  contract  because  they  were  not  the  lowest  bidder. 
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On  the  other  hand,  contract  award  is  often  determined  by  other  factors  in  addition 
to  price  under  competitive  proposal  procedures.  The  specific  number  and  types  of 
evaluation  factors  depend  on  the  nature  of  the  procurement  and  procuring  officials  have 
broad  discretion  in  these  matters.  Contract  award  is  not  conducted  through  public  bid 
openings  and  a  lengthy  period  of  negotiations  and  evaluations  may  transpire  before  an 
award  is  annoimced.  Since  the  award  may  be  based  on  multiple  evaluation  factors  and 
not  just  price  alone,  unsuccessful  offerors  may  not  understand  why  their  firm  was  not 
selected. 

It  is  not  unusual  for  contractors  who  are  interested  in  winning  Government 
contracts  to  invest  a  great  deal  of  time  and  money  in  preparing  a  proposal  in  response  to  a 
Government  Request  for  Proposal  (RFP).  They  analyze  current  and/or  past  contracts, 
conduct  research  to  fully  understand  the  Government’s  requirements,  and  disburse 
significant  resources  in  developing  a  solution  that  represents  the  most  competitive  offer 
possible.  Failure  to  make  such  investments  could  lead  to  a  competitive  disadvantage  in 
the  selection  process.  After  expending  considerable  resources  preparing  their  proposals, 
unsuccessful  bidders  want  to  know  why  they  were  not  awarded  the  contract.  They  may 
feel  that  the  process  was  unfair,  that  there  must  have  been  a  mistake,  or  they  may  want  to 
know  where  their  proposal  was  weak  so  they  may  be  more  competitive  in  future 
procurements.  Failure  to  provide  answers  to  unsuccessful  bidders’  questions  regarding 
the  source  selection  may  result  in  the  filing  of  a  protest. 
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Protests  Exist 


A  protest,  as  formally  defined  by  the  Federal  Acquisition  Reform  Act  of  1994, 
includes  “a  written  objection  to  a  solicitation,  cancellation  of  a  solicitation,  award  or 
proposed  award,  and  termination  or  cancellation  of  an  award  if  improprieties  are  alleged.” 
The  protest  process  is  a  means  of  ensuring  that  the  Government  conducts  procurements  in 
accordance  with  statutory  and  regulatory  procedures.  Additionally,  protests  serve  a  vital 
role  in  assuring  full  and  open  competition  in  the  federal  acquisition  process.  There  are 
generally  two  reasons  industry  contractors  file  protests:  “(1)  to  gain  the  contract;  or  (2)  to 
correct  perceived  inequities  in  the  evaluation,”  [Cooper,  1995:  5].  There  is  little  the 
Government  can  do  to  minimize  protests  filed  for  the  first  reason  without  running  a  risk 
of  making  federal  contracts  less  attractive  and  adversely  affecting  competition.  For 
example,  the  Government  could  reduce  these  type  of  protests  by  making  it  more  difficult 
or  less  attractive  to  file  protests.  However,  this  could  make  Government  contracts  less 
appealing  and  discourage  some  potential  bidders  from  competing.  On  the  other  hand,  the 
Government  can  impact  the  number  of  protests  filed  to  correct  perceived  inequities  in 
evaluations.  The  fact  that  these  are  perceived  to  be  inequities  can  be  removed  or 
prevented  through  an  increase  in  information  flow  between  parties.  This  could  be 
potentially  accomplished  through  more  thorough  and  timely  debriefings  provided  to 
unsuccessful  offerors,  and  identifying  significant  evaluation  factors  and  their  relative 
importance  in  all  solicitations. 

Disappointed  bidders  desiring  to  lodge  a  protest  have  many  forums  in  which  to 
choose  from:  the  awarding  agency  itself,  the  Court  of  Federal  Claims,  U.S.  District 
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Court,  and  the  General  Accounting  Office  (GAO)  [Fausti  and  Lee,  1997:  10].  The 
availability  of  a  particular  forum  to  hear  and  decide  a  protest  is  determined  by  a  number 
of  factors  such  as  the  agency  involved,  the  nature  of  the  protester,  and  the  time  of  the 
protest  [Cibinic  and  Nash,  1986:  1009].  “Where  multiple  forums  are  available,  the 
protestor  faces  a  tactical  choice  based  upon  such  factors  as  the  nature  of  relief  desired,  the 
formality  of  procedures,  the  standards  of  revioAV,  and  the  likelihood  of  obtaining 
substantial  relief,”  [Cibinic  and  Nash,  1986:  1009]. 

This  chapter  will  discuss  the  costs  of  these  protests  to  both  the  contractor  and  the 
Government,  and  what  the  Government  has  done  to  reduce  protests. 

Costs  Of  Protests 

Protests  can  be  costly  in  many  aspects.  The  most  obvious  costs  involve  the  actual 
financial  resources  required  to  complete  the  protest  process.  Attorney  fees, 
administrative  costs,  witness  fees,  and  filing  fees  are  just  a  few  of  the  types  of  direct  costs 
that  may  be  incurred  by  both  parties  during  protests.  Protests  can  be  lengthy,  and 
obviously  incur  greater  costs  as  time  passes.  In  fact  under  current  rules,  the  GAO  can  take 
up  to  100  calendar  days  after  the  filing  of  the  protest  to  render  a  decision  [Fausti  and  Lee, 
1997:  11].  Additionally,  contract  performance  may  be  suspended  during  protests, 
preventing  timely  delivery  and  satisfaction  of  Government  requirements.  When  these 
delays  involve  major  programs,  the  resultant  costs  could  conceivably  amount  to  millions 
of  dollars. 

In  addition  to  the  direct  costs,  there  are  also  indirect  costs  of  protests  which  may 
not  be  as  obvious,  but  are  just  as  significant.  In  the  course  of  a  protest,  there  is  the 
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possibility  of  the  board  or  courts  requiring  a  new  interpretation  of  the  law  in  reaching 
their  decision,  resulting  in  new  precedence.  The  creation  of  new  precedence  is  not 
always  harmful,  however  it  can  potentially  alter  and  complicate  the  procurement  process. 
Protests  can  also  create  public  relations  problems  for  the  Government.  Other  contractors 
may  adopt  the  impression  that  dealing  with  the  Government  is  not  worth  the  hassle  or  that 
the  Government  is  not  fair  in  their  dealings.  This  is  in  addition  to  any  harm  that  may 
come  to  the  relationship  between  the  protesting  contractor  and  the  Government. 
Additionally,  other  bidders  in  the  procurement  may  spend  significant  amounts 
maintaining  “proposal  teams”  or  keeping  open  capacity  in  their  firms  during  a  protest  in 
the  event  that  it  results  in  a  resolicitation.  All  of  these  costs  may  adversely  impact  the 
level  of  competition  in  Government  procurements,  either  directly  or  indirectly. 

The  Government  recognizes  the  negative  impact  protests  have  on  the  procurement 
process.  Consequently,  the  Government  has  enacted  several  laws  and  regulations  in  an 
effort  to  reduce  the  number  of  protests  while  maintaining  the  integrity  of  the  procurement 
system.  Some  of  the  more  significant  laws  include  the  Competition  in  Contracting  Act  of 
1984  (CICA),  the  Federal  Acquisition  Streamlining  Act  of  1994  (FASA),  and  the  Federal 
Acquisition  Reform  Act  of  1996  (FARA).  Each  of  these  acts  have  either  added, 
removed,  or  modified  pre-existing  acquisition  policies  and  regulations,  and  each  has  had 
differing  effects  on  the  frequency  of  bid  protests.  We  will  next  discuss  each  of  the  acts 
and  the  relevant  provisions  that  have,  or  are  expected  to,  impact  bid  protest  frequency. 
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Competition  In  Contracting  Act 

The  Competition  in  Contracting  Act  of  1984  modified  existing  Federal 
procurement  statutes  and  the  GAO  bid  protest  system.  The  intent  of  CICA  was  to 
“increase  competition  in  the  award  of  Government  contracts”  and  to  “eliminate  the 
imbalances  in  the  protest  system,”  [Walsh,  June  1987:  53].  Some  of  the  major  changes 
resulting  from  CICA  are  as  follows  [Walsh,  June  1987:  54-55]: 

■  eliminated  preference  for  formal  advertising  and  put  competitive  negotiation 
on  the  same  level  as  sealed  bid  procedures. 

■  eliminated  the  seventeen  exceptions  to  formal  advertising  and  established 
seven  exceptions  imder  which  “other  than  competitive  procedures”  may  be 
used. 

■  required  sealed  bid  procedures  when  four  specific  conditions  are  met, 
otherwise  competitive  proposals  are  to  be  used. 

Additionally,  CICA  formally  empowered  the  Comptroller  General  (GAO)  to  hear 
protests  and  to  reformat  the  award  protest  system  by  [Walsh,  June  1987:  55]: 

•  directing  the  GAO  to  issue  and  publicize  bid  protest  procedures; 

•  setting  mandatory  time  limits  for  decisions  on  the  merits  of  the  protests; 

•  making  protest  injunctions  virtually  automatic;  this  stay  of  award  prohibits 
contract  award  after  the  contracting  officer  has  received  notice  of  a  protest  to 
the  Comptroller  General  and  while  the  protest  is  pending.  Additionally,  CICA 
mandates  that  contract  performance  be  ceased  or  the  contract  suspended  for 
post-award  protests  filed  within  1 0  days  of  contract  award.  However,  the  head 
of  the  responsible  procurement  activity  can  notify  the  Comptroller  General 
and  authorize  performance  based  on  a  written  finding  that  contract 
performance  will  be  in  the  Government’s  best  interest  and  that  the  urgent  and 
compelling  interests  of  the  United  States  will  not  permit  awaiting  the  decision 
of  the  Comptroller  General. 

•  directing  GAO  to  disregard  intervening  cost  and  performance  factors  in  cases 
where  the  agency  head  overrides  the  suspension  and  allows  contract 
performance  to  continue  while  the  protest  is  pending;  and, 

•  granting  power  to  the  GAO  to  recommend  any  of  several  actions,  including 
monetary  award  to  protesters  to  cover  bid  protest  and  bid  and  proposal  costs. 
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The  intent  of  Congress  in  creating  CICA  was  to  enhance  competition  in 
Government  procurements,  ironically  however  they  may  have  indirectly  invited 
additional  protests.  Requiring  the  GAO  to  advertise  its  new  bid  protest  process  and 
offering  automatic  stays  and  terminations  could  have  led  potential  protestors  to  believe 
that  there  was  a  more  lenient  basis  for  protests.  Previous  research  on  the  impact  of  CICA 
on  the  number  of  protests  indicated  that  there  was  a  10-15%  increase  in  protests  in  the 
years  following  CICA’s  enactment  [Walsh,  June  1987:  130]. 

CICA  was  effective  in  certain  aspects  of  increasing  competition  in  Government 
procurement.  However,  the  undesired  effect  of  increasing  protests  was  unacceptable. 
Further  studies,  laws  and  regulations  were  enacted  in  an  attempt  to  counter  the  negative 
effects  of  CICA  and  streamline  the  acquisition  process.  One  such  act,  the  National 
Defense  Authorization  Act  of  1991,  directed  the  Department  of  Defense  to  establish  an 
advisory  panel  of  experts  to  review  all  laws  affecting  procurement.  This  panel  became 
known  as  the  Section  800  panel  and  recommended  changes  in  over  600  statutes  affecting 
Government  procurement  and  defense  acquisition.  These  recommendations  became  the 
basis  for  the  changes  included  in  the  Federal  Acquisition  Streamlining  Act  of  1994 
[Cooper,  June  1995:  14].  The  next  section  discusses  the  implementation  of  the  Federal 
Acquisition  Streamlining  Act  and  identifies  some  of  its  significant  provisions  as  they 
pertain  to  bid  protests. 

The  Federal  Acquisition  Streamlining  Act 

The  Federal  Acquisition  Streamlining  Act  was  signed  into  effect  on  October  13, 

1 994  and  made  significant  changes  to  the  acquisition  process  by  eliminating  or  modifying 
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over  225  existing  statutes.  The  act  made  significant  changes  in  the  following  areas  of 
acquisition:  task  and  delivery  order  contracting,  advisory  and  assistance  services 
contracting,  source  selection  procedures  and  debriefings,  bid  protest  procedures, 
contractor  past  performance  and  other  evaluation  factors,  new  micro  purchase  procedures, 
commercial  item  procurement,  small  business  awards,  and  several  other  areas.  The 
themes  behind  the  changes  made  by  FASA  are  a  preference  for  moving  to  commercial 
contracting  methods,  transitioning  the  procurement  process  to  an  electronic  basis, 
eliminating  paperwork  burdens  in  the  procurement  cycle,  and  eliminating  non-value 
added  requirements.  The  following  is  a  summary  of  the  major  changes  enacted  by  FASA 
[Kidd,  1996:  26-30],  [the  Federal  Acquisition  Streamlining  Act  of  1994]: 

•  Allows  the  use  of  a  credit  card  method  for  “micro  purchases  (procurements  up 
to  $2,500)”  with  virtually  no  paperwork  or  oversight  required  for  each 
individual  purchase. 

•  Simplified  acquisition  threshold  (procurements  from  $2,500  to  $50,000  and, 
after  FACNET  certification  to  $100,000)  which  reduces  paperwork  and  moves 
DOD  to  electronic  procurement  procedures. 

•  Allows  the  use  of  contractor’s  past  performance  as  a  major  evaluation  factor 
in  the  source  selection  process. 

•  Requires  agencies  to  limit  the  number  of  proposals  in  the  competitive  range  to 
provide  for  efficient  competition  among  the  highest  rated  competitors. 

Another  area  addressed  by  FASA  is  the  debriefing  process.  Under  FASA, 
agencies  are  now  required  to  provide  a  debriefing  to  any  interested  offeror  as  to  why  they 
were  excluded  from  the  competitive  range  or  from  further  consideration,  provided  the 
offeror  submits  a  written  request  for  debriefing  within  three  days  of  learning  of  their 
exclusion  [Acquisition  Streamlining  Institute,!  997:  32].  There  are  a  number  of  reasons 
supporting  the  need  for  briefing  unsuccessful  offerors.  Considerations  of  fairness  dictate 
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such  communication  in  response  to  the  effort  that  an  offeror  has  put  into  preparing  and 
submitting  the  proposal  [Cibinic  and  Nash,  1986:  655].  It  may  also  be  beneficial  to  the 
Government  to  thoroughly  inform  the  offeror  of  their  proposal’s  deficiencies  so  that  the 
same  mistakes  are  not  repeated  in  subsequent  proposals.  By  avoiding  the  same  mistakes 
and  improving  their  proposals,  contractors  become  more  capable  of  responding  to  the 
Government’s  requirements  in  future  procurements.  Consequently,  comf)etition  is 
enhanced,  allowing  the  Government  to  not  only  have  its  needs  more  effectively  satisfied, 
but  often  at  a  better  price.  Providing  useful  debriefing  information  can  ultimately  reduce 
the  cost  of  the  competitive  process  and  increase  the  likelihood  that  offerors  will  continue 
to  view  the  Government  market  as  a  worthwhile  place  to  invest  its  resources  as  well. 

“For  many  years,  contractors  have  complained  that  debriefings  were  not  serving 
their  intended  purpose,”  [Acquisition  Streamlining  Institute,  1997:  24].  “To  be  effective 
in  their  intended  purpose,  debriefings  should  inform  the  offeror  why  it  was  not  awarded 
the  contract  in  order  to  improve  subsequent  proposals,  thereby  furthering  full  and  open 
competition,”  [Acquisition  Streamlining  Institute,  1997:  24].  However,  a  lack  of  guidance 
coupled  with  a  fear  of  revealing  proprietary  or  source  selection  information  resulted  in 
Government  personnel  being  reluctant  to  divulge  significant  information  about  the 
procurement.  Consequently,  companies  were  limited  in  their  options  for  discovering  the 
basis  of  an  award  decision  and  often  resorted  to  filing  bid  protests  [Acquisition 
Streamlining  Institute,  1997:  24].  In  an  effort  to  break  such  a  cycle  and  help  “create  a 
government  that  works  better  and  costs  less,”  Congress  included  the  following  sections  in 
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FASA,  pertaining  to  post-award  debriefs  protests  [Acquisition  Streamlining 
Institute,!  997:  33]: 

•  When  a  contract  is  awarded  by  the  head  of  an  agency  on  the  basis  of 
competitive  proposals,  an  unsuccessful  offeror,  upon  written  request  received 
by  the  agency  within  3  days  after  the  date  on  which  the  imsuccessftil  offeror 
receives  the  notification  of  the  contract  award,  shall  be  debriefed  and 
furnished  the  basis  for  the  selection  decision  and  contract  award.  The  head  of 
the  agency  shall  debrief  the  offeror  within,  to  the  maximum  extent  practicable, 
five  days  after  receipt  of  the  request  by  the  agency. 

•  The  debriefing  shall  include,  at  a  minimum- 

-  the  agency’s  evaluation  of  the  significant  weak  or  deficient  factors  on 
the  offeror’s  offer; 

-  the  overall  evaluated  cost  and  technical  rating  of  the  offer  of  the 
contractor  awarded  the  contract  and  the  overall  evaluated  cost  and 
technical  rating  of  the  offer  of  the  debriefed  offeror; 

-  the  overall  ranking  of  all  offers; 

-  a  summary  of  the  rationale  for  the  award; 

-  in  the  case  of  a  proposal  that  includes  a  commercial  item  that  is  an  end 
item  under  the  contract,  the  make  and  model  of  the  item  being  provided 
in  accordance  with  the  offer  of  the  contractor  awarded  the  contract;  and 

-  reasonable  responses  to  relevant  questions  posed  by  the  debriefed  offeror 
as  to  whether  source  selection  procedures  set  forth  in  the  solicitation, 
applicable  regulations,  and  other  applicable  authorities  were  followed  by 
the  agency. 

•  The  debriefing  may  not  include  point  -by-point  comparisons  of  the 
debriefed  offeror’s  offer  with  other  offers  and  may  not  disclose  any 
information  that  is  exempt  from  disclosure. 

“The  scope  of  the  information  that  FASA  mandates  agencies  provide  in  debriefs  is 
rather  broad  and  intended  to  reveal  sufficient  details  about  the  competition  to  permit  an 
offeror  to  understand  the  rationale  for  the  agency’s  award  decision,”  [Acquisition 
Streamlining  Institute,  1997:  24].  The  importance  of  thorough  debriefings  was 
emphasized  during  a  past  report  of  the  Senate  Committee  on  Governmental  Affairs 
[Senate  Report  No.  103-258, 103’“^  Congress,  session  7, 1994]: 
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The  inability  to  obtain  sufficient  information  in  debriefing,  in  particular  a 
meaningful  explanation  of  the  basis  for  the  award  decision,  leads  many 
firms  to  file  protests  in  order  to  obtain  that  information. 

The  primary  purpose  of  mandating  meaningful  debriefings  as  part  of  the 
acquisition  streamlining  effort  is  to  eliminate  the  filing  of  protests  as  a 
means  to  discover  the  propriety  of  an  award  decision. 

FASA  mandates  certain  minimum  information  be  provided  to  unsuccessful 

bidders  during  debriefs,  however  Contracting  Officers  still  have  the  latitude  to  provide 

additional  information  at  their  discretion,  particularly  if  this  supplementary  information 

might  prevent  a  protest.  Certain  information  is  still  prohibited  from  release  under 

existing  regulations,  however.  The  Freedom  of  Information  Act,  which  states  that  it  does 

not  authorize  the  withholding  of  any  information  otherwise  required  to  be  disclosed  by 

law,  and  FAR  15.1004(e)  still  preclude  revealing  trade  secrets,  the  names  of  individuals 

providing  reference  information  about  the  contractor’s  past  performance,  any  information 

which  is  not  relevant  to  the  award  process,  or  other  privileged  and  confidential 

information  [Acquisition  Streamlining  Institute,  1997:  26].  In  addressing  this  issue. 

Congress  stated  that  agencies  should  withhold  information 

relating  to  trade  secrets;  privileged  or  confidential  manufacturing 
processes  and  techniques;  and  commercial  and  financial  information  that 
is  privileged  or  confidential,  including  cost  breakdowns,  profit,  indirect 
cost  rates,  and  similar  information.  [Acquisition  Streamlining  Institute, 
1997,  p.  26] 

In  addition  to  addressing  the  content  of  debriefs,  FASA  also  changed  the 
procedures  for  requesting  and  responding  to  requests  for  debriefs.  Prior  to  FASA’s 
implementation,  agencies  were  required  to  conduct  debriefings  “as  soon  as  possible”  after 
contract  award.  The  provisions  of  FASA  now  mandates,  however  that  agencies  must 
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notify  unsuccessful  offerors  of  an  award  within  three  days  of  the  award.  Offerors  then 
have  the  option  to  request  a  debrief  as  long  as  they  do  it  within  three  days  of  receiving  the 
award  notification.  Agencies  are  required  to  respond  to  requests  for  debriefs  and  conduct 
a  debriefing  within  five  days  of  receiving  the  request.  Agencies  may,  at  their  discretion, 
accommodate  late  requests  for  debriefs,  however  they  are  not  required  to  do  so.  This  is  a 
departure  from  past  practices  where  every  offeror  was  entitled  to  a  debriefing  regardless 
of  when  it  was  filed. 

Another  area  addressed  by  FAS  A  that  could  impact  the  frequency  of  bid  protests 
is  evaluation  criteria.  FAR  §  15.605,  regarding  evaluation  of  proposals  in  a  negotiated 
procurement  was  significantly  altered  by  FASA.  “The  intent  of  these  modifications  was 
to  result  in  solicitations  that  were  more  definite  in  regards  to  evaluation  criteria,  and  in 
more  awards  based  on  initial  proposals,”  [Acquisition  Streamlining  Institute,  1997:  19]. 
Previously,  there  were  occurrences  of  agencies  listing  the  relative  importance  of  the 
technical  factors  in  procurements  but  failing  to  state  how  these  factors  related  to  price 
[Acquisition  Streamlining  Institute,  1997,  p.  20].  In  such  cases,  the  GAO  has  held  that 
when  an  agency  does  not  explicitly  state  the  relative  importance  of  the  cost  and  technical 
factors  “it  must  be  presumed  that  cost  and  technical  considerations  will  be  considered 
approximately  equal  in  weight,”  [Riggins  Co.,  Inc.,  B-244460,  84-2  CPD  ^  137]. 
Subsequent  to  FASA,  agencies  must  now  identify  in  their  solicitations  all  significant 
evaluation  factors  in  the  source  selection  and  their  relative  importance.  Additionally,  the 
solicitation  is  required  to  include  whether  the  non-cost  factors  are  “significantly  more 
important,”  “approximately  equal  to,”  or  “significantly  less  importanf  ’  than  cost. 
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Contracting  Officers  may,  at  their  discretion,  provide  additional  details  to  clarify  the 
relative  importance  of  the  factors,  however  they  are  not  required  to  do  so. 

Not  all  of  the  provisions  of  FAS  A  may  have  the  desired  effect  of  reducing  bid 
protests.  Some  maintain  that  FASA’s  increased  emphasis  on  the  use  of  contractors  past 
performance  as  an  evaluation  factor  could  potentially  result  in  an  increase  in  protests 
[interview,  2  Apr  97].  FAS  A  established  the  requirement  for  all  Government  agencies  to 
collect  and  maintain  information  on  contract  performance  and  required  the  Office  of 
Procurement  Policy  (OFPP)  to  establish  guidance  for  agencies  pertaining  to  the 
evaluation  of  past  performance.  OFPP  amended  FAR  parts  9,15,  and  42  to  impose  a 
number  of  requirements  [Acquisition  Streamlining  Institute,  1997,  pp.  21-22]: 

•  All  contractors  must  have  a  “satisfactory  performance  record”  (unless  it  has  no 
record  at  all),  FAR  9. 104- 1(c); 

•  Contracting  officers  must  consider  past  performance  when  making  a 
determination  of  a  contractor’s  responsibility,  FAR  9. 105- 1(c); 

•  Contracting  officers  are  directed  to  acquire  data  on  contractor  past 
performance  from  a  myriad  of  sources,  both  government  and  nongovernment; 

•  A  new  FAR  Subpart  42. 1 5,  Contractor  Performance  Information,  specifies 
that  an  agency  must  retain  past  performance  information,  permit  contractors  to 
submit  rebuttal  information,  and  share  such  information  with  other  agencies 
for  a  period  of  not  more  than  three  years  after  completion  of  contract 
performance. 

•  Requires  that  the  relationship  of  past  performance  and  other  non-price  factors 
be  clearly  stated  in  the  solicitation. 

Additionally  the  regulations  mandate  that  past  performance  be  used  as  an 
evaluation  factor  in  all  competitively  negotiated  procurements  valued  over  a  certain 
threshold  determined  by  the  date  of  issuance  [Acquisition  Streamlining  Institute,  1997; 
22]. 
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In  addition  to  the  determination  of  responsibility,  FAR  15.608(a)(2)(i)  mandates 
that  the  agency  evaluate  the  “number  and  severity  of  an  offeror’s  problems,  the 
effectiveness  of  corrective  actions  taken,  the  offeror’s  overall  work  record,  and  the  age 
and  relevance  of  past  performance  of  information.”  The  information  considered  shall 
only  include  Federal,  state,  and  local  governments  of  a  similar  nature,  as  well  as  any  other 
information  from  known  sources  [FAR  15.608(a)(2)(ii)].  If  past  performance  information 
for  a  particular  firm  is  lacking  or  imavailable,  the  firm  will  receive  a  “neutral”  evaluation 
in  that  area  [FAR  15.608(a)(2)(iii)].  As  mentioned  earlier,  FAS  A  mandates  that  agencies 
disclose  certain  information  at  post  award  debriefs,  however,  FAR  §  15.1004(e)(4) 
prohibits  the  release  of  names  of  individuals  providing  reference  information  about  an 
offeror’s  past  performance.  This  undisclosed  information  could  contribute  to 
unsuccessful  bidders  protesting  an  award  decision  if  they  feel  that  the  past  performance 
data  is  inaccurate  or  provided  by  a  source  with  malicious  intent  [Cooper,  1994;  5]. 
Additionally,  this  prohibition  could  deprive  an  offeror  unfairly  criticized  from  adequately 
addressing  perceived  performance  shortcomings. 

Some  contracting  officers  have  offered  the  recent  source  selection  of  the  Joint 
Primary  Aircraft  Training  System  (JPATS)  to  illustrate  the  point  that  FASA  may  actually 
increase  protest  frequency.  They  emphasized  that  despite  abiding  by  all  implemented 
FASA  regulations  and  guidelines,  the  JPATS  became  one  of  the  largest  programs  to  ever 
be  protested  [interview,  2  Apr  97].  The  $7-billion  program  was  intended  to  replace  the 
Air  Force’s  aging  T-37B  and  the  Navy’s  T-34C  as  training  aircraft.  When  it  was 
announced  that  Raytheon  was  the  winner  of  the  competition  in  June  1995,  Cessna  and 
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Rockwell  filed  protests  over  the  Air  Force’s  definition  of  best  value  and  the  use  of 
unannounced  criteria  in  the  competition  [Fulghum,  July  24,  1995;  24],  Both  companies 
complained  that  they  were  misled  into  believing  that  the  competition  was  for  best  value, 
rather  than  lowest  cost.  As  mentioned  previously,  FASA  does  specifically  address  the 
inclusion  of  significant  evaluation  factors  that  will  be  considered  in  making  the  source 
selection  and  their  relative  importance.  This  would  seem  to  contradict  statements  that 
this  particular  source  selection  fully  complied  with  FASA.  However,  according  to  senior 
Air  Force  acquisition  officials,  “the  inability  of  the  Air  Force  to  quantify  or  define  best 
training  value  led  acquisition  officials  to  select  the  lowest-unit-priced  Raytheon  Mk.2  as 
the  winner  of  the  competition,”  [Fulghum,  Feb  12,  1996:  60].  The  unnamed  official  went 
on  to  say  that  “Rockwell’s  aircraft  was  the  most  advanced  technologically  and  Cessna’s 
lifecycle  cost  was  actually  a  little  lower,  but  because  no  one  could  actually  quantify  best 
training  value,  the  Air  Force  went  with  the  lowest  initial  acquisition  cost,”  [Fulghum,  Feb 
12,  1996:  60].  The  GAO  reviewed  both  the  selection  process  and  the  Government’s 
choice  of  Raytheon  and  subsequently  rejected  Cessna  and  Rockwell’s  protests  on  the 
grounds  that  the  selection  process  was  not  improper  [Fulghum,  Feb  12, 1996:  60].  This 
decision  implies  that  the  allegations  were  perhaps  unfounded  and  that  the  competition 
was  in  compliance  with  applicable  provisions  of  FASA.  A  finding  for  the  protestors  in 
this  instance  would  have  implied  that  the  agency  had  not  complied  with  FASA,  however 
this  was  not  the  case. 

The  next  question  is  whether  the  agency’s  compliance  with  FASA  contributed  to 
the  protest.  The  grounds  for  the  protest  and  the  GAO’s  decision  in  this  case  suggests  just 
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the  contrary.  In  fact,  the  protestors  allegations  of  unannounced  evaluation  criteria  is 
specifically  addressed  by  FASA  as  mentioned  previously.  Therefor,  it  does  not  appear 
that  FASA  contributed  to  this  protest,  however  it  is  possible  that  perhaps  this  area  was 
not  adequately  addressed  in  the  Act. 

Implementation  of  FASA 

Implementing  FASA  was  a  complex  process  involving  revisions  to  numerous 
regulations,  principally  the  Federal  Acquisition  Regulation  (FAR).  Revisions  were  also 
needed  in  the  Defense  Acquisition  Regulation  Supplement  (DFARS),  other  agencies  FAR 
supplements,  and  miscellaneous  other  regulations.  Additionally,  FASA  resulted  in  the 
issuance  of  nonregulatory  guidance  such  as  DOD  directives  and  instructions.  Although 
the  Act  makes  sweeping  changes  in  a  number  of  areas,  many  of  these  changes  had  little 
impact  on  the  contracting  process  at  certain  agencies.  While  President  Clinton 
proclaimed  FASA  as  the  most  significant  piece  of  procurement  reform  legislation  since 
the  passage  of  CICA  in  1984,  many  others  have  criticized  FASA  for  not  going  far  enough 
[Acquisition  Streamlining  Institute,  1997,  p.  13].  Some  critics  maintain  that  more  should 
have  been  done  in  other  areas,  such  as  those  aimed  at  reducing  the  frequency  of  bid 
protests.  In  Air  Force  Material  Command  (AFMC),  the  Air  Force’s  largest  buying 
organization,  FASA  had  little  impact  on  the  way  they  do  business  in  certain  respects. 
Some  contracting  officers  and  acquisition  professionals  maintain  that  FASA  had  little 
impact  on  AFMC’s  contracting  process,  particularly  as  it  relates  to  the  areas  previously 
discussed  as  being  critical  to  protest  frequency,  including  debriefs  [interview,  4  Apr  97]. 
They  assert  that  the  practices  within  their  organizations  either  met  or  surpassed  the 
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requirements  implemented  by  FASA,  or  that  somehow,  particular  provisions  of  the  Act 
were  not  applicable  to  their  organizations.  One  interviewee  stated  that  the  only  FASA- 
related  changes  in  her  organization  (ASC)  that  might  have  an  impact  on  protest  frequency 
was  the  addition  of  past  performance  as  an  evaluation  factor  and  the  requirement  to 
provide  debriefs  within  three  days  of  their  request  [interview,  2  Apr  97].  Other  than  these 
areas,  it  appears  that  FASA  had  little  formal  impact  on  these  two  organization’s 
contracting  processes  and  may  not  have  a  significant  impact  on  the  frequency  of  protests. 
The  Federal  Acquisition  Reform  Act  of  1996 

One  of  the  most  recent  attempts  at  the  improvement  of  the  Federal  acquisition 
process  is  the  Federal  Acquisition  Reform  Act  (FARA).  The  changes  imposed  by  FARA 
should  be  seen  as  further  adjustments  that  complement  the  reforms  made  by  FASA.  The 
final  rule,  that  became  effective  on  January  1, 1997,  amended  the  FAR  to  require  that 
contracting  officers,  prior  to  contract  award,  provide  a  debriefing  to  an  interested  offeror 
on  the  reasons  for  that  offeror’s  exclusion  from  the  competitive  range  [Government 
Contracts  Alert,  1997:1].  Pursuant  to  this  new  rule,  the  contracting  officer  may  refuse  to 
provide  a  pre-award  debriefing  request  if :  (1)  it  is  not  in  the  best  interests  of  the 
Government,  at  the  time  the  debriefing  is  requested,  to  provide  such  a  debriefing 
(however,  the  contracting  officer  must  provide  the  requested  debriefing  no  later  than  the 
time  at  which  the  agency  conducts  its  post-award  debriefings);  or,  (2)  a  written  request 
from  the  offeror  for  such  a  debriefing  is  not  received  by  the  contracting  officer  within 
three  days  of  the  offeror’s  receipt  of  a  notice  of  exclusion  from  the  competitive  range 
[Government  Contracts  Alert,  1997:1]. 
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Under  the  new  rule,  pre-award  debriefings  may  be  performed  orally  or  in  writing, 
and  at  a  minimum,  should  include  the  following  information:  (1)  the  agency’s  evaluation 
of  significant  elements  in  the  offeror’s  proposal;  (2)  a  summary  of  the  rationale  for 
excluding  the  offeror  from  the  competitive  range;  and,  (3)  reasonable  responses  to 
relevant  questions  about  whether  source  selection  procedures  contained  in  the 
solicitation,  applicable  regulations,  and  other  applicable  authorities  were  followed  in  the 
process  of  excluding  the  offeror  from  the  competitive  range.  Pre-award  debriefings 
should  not  disclose  (1)  the  niunber  of  offerors;  (2)  the  identity  of  the  other  offerors;  (3) 
the  content  of  the  other  offerors’  proposals;  (4)  the  ranking  of  the  other  offerors;  (5)  the 
evaluation  of  the  other  offerors;  or,  (6)  any  of  the  information  prohibited  from  disclosure 
during  post-award  debriefings.  Additionally,  FARA  established  civil  and  criminal 
penalties  for  the  unauthorized  disclosure  of  contractor  bid  or  proposal  information,  or 
source  selection  information,  before  the  award  of  the  procurement  contract  to  which  the 
information  relates. 

These  three  acts  are  among  the  more  significant  efforts  of  the  United  States 
Government  to  reform  acquisition  and  reduce  post-award  protests.  Were  they  successful 
in  their  efforts?  The  following  section  will  discuss  the  author’s  propositions  for  this 
study  in  an  attempt  to  provide  more  insight  into  their  success. 

Propositions 

The  intent  of  this  research  is  to  discern  the  effectiveness  of  Government  efforts  at 
reducing  post-award  protest  frequency.  As  mentioned  earlier,  CICA  made  filing  protests 
easier  and  a  previous  study  indicated  that  protests  actually  increased  10-15%  over  the 
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following  years.  FARA  just  went  into  effect  on  1  January  1997,  so  it  would  be  premature 
to  study  its  effects  due  to  lack  of  data.  FAS  A  on  the  other  hand  has  been  in  effect  for 
over  two  years  at  the  time  of  this  writing  and  sufficient  data  should  be  available  to 
conduct  a  preliminary  analysis  of  any  effects  it  may  have  had  on  protest  frequency. 
Known  previous  research  had  considered  FASA  in  its  analysis  of  the  Army’s  installation 
level  post-award  debriefing  process  and  included  recommendations  for  improvement.  At 
the  time  of  this  writing  however,  there  did  not  appear  to  be  any  known  research  on 
FASA’s  effects  on  the  frequency  of  protests  or  whether  offerors’  proposals  have 
improved  as  a  result. 

As  previously  mentioned,  there  are  generally  two  reasons  why  industry  files 
protests:  (1)  to  gain  the  contract;  or  (2)  to  correct  perceived  inequities  in  the  evaluation. 
By  introducing  more  stringent  requirements  for  the  administration  and  content  of  post¬ 
award  debriefs,  and  mandating  the  identification  of  all  significant  evaluation  factors  in 
solicitations,  FASA  attempted  to  reduce  protests  by  addressing  the  second  reason.  While 
these  measures  may  appear  to  be  very  practical  attempts  at  reducing  protests,  it  is  difficult 
to  determine  whether  they  have  had  their  desired  effect.  Based  on  the  information 
discussed  in  this  chapter,  the  author  contrived  the  following  propositions  to  be  the  focus 
of  this  research: 

Proposition  #1 :  FASA  has  not  impacted  the  United  States  Air  Force’s  source 
selection  processes  in  a  manner  that  would  significantly  affect  post-award  protest 
frequency.  Preliminary  unstructured  interviews  were  conducted  with  contracting  officers 
in  the  policy  offices  of  both  AFMC  and  its  subordinate  command  AS C.  The  central 
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theme  of  the  responses  seemed  to  indicate  that  a  majority  of  the  provisions  that  the  author 
concluded  to  be  relevant  to  reducing  protests  were  being  practiced  by  their  organizations 
prior  to  FASA’s  enactment.  The  relevant  provisions  being  referred  to  are  those  that 
pertain  to  the  content  and  administration  of  post-award  debriefs,  and  identification  of  all 
significant  evaluation  factors  in  solicitations.  In  order  to  test  this  proposition,  the  author 
will  conduct  structured  telephone  interviews  with  contracting  officers  in  AFMC  and  its 
subordinate  commands,  ASC,  SPC,  and  ESC.  AFMC  was  chosen  as  the  unit  of  analysis 
in  this  study  because  it  manages  approximately  52%  of  the  annual  United  States  Air 
Force  budget  and  represents  the  largest  buying  organization  in  that  service  IWright- 
Patterson:  Legacy.  1995;  17].  Results  from  the  interview  should  be  acceptable  for 
generalization  to  the  Air  Force  as  a  whole  since  AFMC  constitutes  such  a  large 
proportion  of  the  Air  Force’s  acquisition  activity.  The  primary  research  question  to  arise 
as  a  result  of  this  proposition  is  “How  did  FASA  impact  Air  Force  source  selection 
procedures  as  they  relate  to  content  and  administration  of  post-award  debriefs,  and 
evaluation  criteria  practices.”  Refer  to  chapter  3  for  details  on  the  content  of  the 
interview  and  how  it  will  be  used  to  assay  proposition  #1 . 

Proposition  #2:  FASA  has  not  impacted  post-award  protest  frequency  within  the 
United  States  Air  Force.  FASA  was  intended  to  reduce  protests  by  introducing  more 
stringent  requirements  for  the  administration  and  content  of  post-award  debriefs,  and 
mandating  the  identification  of  all  significant  evaluation  factors  in  solicitations.  In  order 
for  these  measures  to  have  an  impact  on  protest  frequency  however,  they  would  have  to 
change  pre-existing  contracting  policy  and  procedures.  While  the  entire  Executive 
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Branch  of  the  United  States  Government  is  subject  to  the  Act,  it  affected  individual 
agencies  differently  due  to  variances  in  pre-existing  policies.  The  author  maintains,  as 
stated  in  proposition  #1,  that  FASA  did  not  significantly  impact  the  Air  Force’s  source 
selection  process.  Therefor,  it  would  seem  logical  that  FASA  did  not  impact  post-award 
protest  frequency  within  the  Air  Force.  To  test  proposition  #2  and  identify  any  trends  or 
statistically  significant  differences,  the  number  of  annual  protests  within  the  Air  Force 
will  be  compared  between  pre-  and  post-FASA  enactment.  The  primary  research 
question  to  arise  as  a  result  of  this  proposition  is  “What  impact  did  FASA  have  on  protest 
frequency?”  Refer  to  chapter  3  for  further  details  on  the  statistical  methods  employed  in 
this  analysis. 

Proposition  #3:  FASA  has  not  impacted  the  quality  of  proposals  submitted  by 
potential  offerors  in  response  to  Air  Force  solicitations.  Proponents  of  FASA 
maintained  that  a  side  benefit  of  the  more  thorough  debriefs  mandated  by  the  Act  would 
result  in  better  quality  proposals  in  the  future.  They  believed  that  providing  the  added 
information  would  allow  the  unsuccessful  offerors  to  learn  from  their  mistakes  and 
inadequacies,  and  submit  proposals  in  the  future  that  would  be  more  responsive  to 
Government  solicitations.  As  previously  stated  in  proposition  #1,  the  author  believes  that 
FASA  did  not  significantly  alter  the  Air  Force’s  source  selection  process.  Preliminary 
indications,  based  on  discussions  with  AFMC  personnel,  are  that  FASA  did  not  impact 
the  content  or  administration  of  the  post-award  debriefs  conducted  by  AFMC.  To  test 
this  proposition,  the  same  case  study  methods  applied  in  testing  proposition  #lwill  be 
used.  The  structured  interview  conducted  for  proposition  #1  will  also  contain  questions  to 
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ascertain  whether  contracting  officers  within  AFMC  and  its  subordinate  commands  have 
witnessed  an  improvement  in  the  quality  of  proposals  since  the  implementation  of 
FASA.  The  primary  research  question  to  arise  as  a  result  of  this  proposition  is  “How  did 
FASA  affect  the  quality  of  candidate  bids?”  Quality  of  proposals  will  be  defined  as 
responsiveness  to  Government  requirements  and  solicitations.  For  further  details  on  the 
content  of  the  interview  and  how  it  will  be  used  to  measure  this  proposition,  refer  to 
chapters. 

Summary 

This  chapter  identified  and  discussed  the  costs  of  protests,  the  importance  of  post¬ 
award  debriefs,  and  some  of  the  Government’s  efforts  at  minimizing  protest  frequency. 
This  discussion  leads  to  the  question  of  whether  the  Government’s  efforts  have  been 
effective  in  meeting  their  goal  of  reducing  the  frequency  of  protests.  Prior  research  has 
indicated  that  CICA  made  filing  protests  easier  and  that  protests  actually  increased  10- 
15%  over  the  following  years  as  a  result.  FARA  just  went  into  effect  on  1  January  1997, 
so  it  would  be  premature  at  the  time  of  this  writing  to  study  its  effects,  due  to  lack  of  data. 
FASA  on  the  other  hand  has  been  in  effect  for  over  two  years  at  the  time  of  this  writing 
and  sufficient  data  should  be  available  to  conduct  a  preliminary  analysis  of  any  effects  it 
may  have  had  on  protest  frequency.  Known  previous  research  had  considered  FASA  in 
its  analysis  of  the  Army’s  installation  level  post-award  debriefing  process  and  included 
recommendations  for  improvement.  At  the  time  of  this  writing  however,  there  did  not 
appear  to  be  any  known  research  on  FASA’s  effects  on  the  fi-equency  of  protests  or 
whether  offerors’  proposals  have  improved  as  a  result. 
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To  provide  further  insight  into  the  effectiveness  of  Government  actions  in  this 
area,  the  author  developed  three  propositions  as  the  focus  of  this  research.  The 
propositions  are  related  to  the  impact  of  FASA  on  Air  Force  source  selection  processes 
and  frequency  of  post-award  protests  within  that  agency. 

In  order  for  the  provisions  of  FASA  to  have  an  impact  on  protest  frequency,  they 
would  have  to  change  pre-existing  contracting  policy  and  procedures.  Some  maintain 
that  this  was  not  the  case  and  that  existing  contracting  practices  were  not  modified  as  a 
result  of  the  Act’s  implementation.  They  assert  that  the  above  mentioned  measures  were 
already  being  practiced  in  their  organization  prior  to  the  passage  of  FASA.  Critics  of  the 
Act  also  contend  that  FASA’s  requirement  to  utilize  past  performance  as  an  evaluation 
factor  could  contribute  to  an  increase  in  protest  frequency. 

Based  on  the  evidence  uncovered  during  the  literature  review,  the  author  proposes 
that  FASA  has  had  negligible  affect  on  actual  contracting  practices  and  consequently  has 
neither  increased  nor  decreased  protest  frequency.  Additionally,  since  it  is  believed  that 
the  Act  had  little  effect  on  actual  contracting  practices,  the  author  also  proposes  that  the 
quality  of  candidate  bid  proposals  has  remained  unchanged.  The  next  chapter  will 
discuss  in  detail,  the  procedures  the  author  will  employ  to  test  these  propositions. 
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III.  Methodology 


General 

Chapter  III  describes  the  research  procedures  used  to  meet  the  research  objectives 
and  to  test  the  propositions  posed  in  Chapter  II.  First,  this  chapter  describes  those 
methods  used  in  an  attempt  to  secure  detailed  data  on  the  frequency  of  bid  protests  within 
the  Air  Force.  Second,  this  chapter  outlines  the  structured  interview  conducted, 
including;  1)  the  sample  population;  2)  selection  of  the  sample  size;  3)  interview 
preparation  and  content;  and,  4)  statistical  methods  used  to  analyze  the  survey  data. 
Information  for  this  thesis  was  collected  through  a  literature  review,  structured  interviews 
with  contracting  officers,  published  and  unpublished  GAO  data,  and  published  and 
unpublished  Air  Force  Material  Command  data. 

Discussion  of  Variables 

Major  Constructs.  The  following  constructs  are  contained  in  the  research  design  and 
warrant  definition: 

1 .  Protest:  A  protest,  as  formally  defined  by  the  Federal  Acquisition  Reform  Act 
of  1994,  includes  a  written  objection  to  a  solicitation,  cancellation  of  a 
solicitation,  award  or  proposed  award,  and  termination  or  cancellation  of  an 
award  if  improprieties  are  alleged.  For  the  purposes  of  this  study,  we  will 
concentrate  only  on  the  number  of  protests  filed  in  response  to  objections  to 
solicitations,  awards,  or  proposed  awards  within  AFMC. 
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2.  Quality  of  Proposals:  Quality  of  proposals  will  be  defined  as  responsiveness 
to  Government  solicitations  and  deals  with  the  question  of  whether  the  offeror 
has  promised  to  do  exactly  what  the  Government  has  requested.  A  key 
element  of  the  sealed  bid  system  is  that  non-responsive  bids  may  not  be 
considered  by  contracting  officers  and  must  be  rejected.  This  requirement  is 
derived  from  the  statutory  provision  that  provides  that  award  be  made  to  the 
bidder  whose  bid  conforms  to  the  solicitation,  10  U.S.C.  §  2305(b)(3),  41 
U.S.C.  §  253b  (c)  [Cibinic  and  Nash,  August  1986:  394].  Candidate  bids  must 
provide  sufficient  information  to  allow  the  contracting  officer  to  determine 
whether  the  overall  bid  is  responsive.  Proposals  should  contain  the  requisite 
supporting  data,  information,  documentation,  certification,  etc. 

3.  Bid  Protests:  This  is  a  measure  of  the  number  of  annual  bid  protests 

filed  against  the  Air  Force  as  a  proportion  of  the  number  of  solicitation  awards 
within  that  agency  that  same  year. 

4.  Changes  In  Source  Selection  Practices:  This  refers  to  changes  in  actual 
contracting  practices  to  occur  as  a  result  of  FASA’s  implementation. 

More  specifically,  this  study  focuses  on  AFMC’s  source  selection 
policies  related  to  the  administration  and  content  of  post-award 
debriefs  and  evaluation  criteria. 

5.  Changes  In  Quality  Of  Candidate  Proposals:  Pertains  to  the  completeness 
and  responsiveness  to  all  requirements  of  the  solicitation. 
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Constructs  To  Be  Tested.  The  variables  used  in  this  design  were  derived  from  the 


literature  review  and  the  author’s  judgment  of  the  factors  important  to  test  the 
propositions.  The  following  are  the  constructs  to  be  tested  in  investigating  the 
propositions: 

1 .  Debrief  Content  -  The  practice  of  providing  an  increased 
amount  of  information  during  post-award  debriefs  to  imsuccessful 
offers.  The  information  of  interest  in  this  study  pertains  to  information 
regarding  reasons  why  unsuccessful  offerors  were  excluded  from  further 
consideration,  information  regarding  weaknesses  in  unsuccessful  offeror’s 
proposals,  and  providing  the  overall  ranking  of  all  offerors. 

2.  Responsiveness  -  The  practice  of  providing  post-award  debriefs  to 
unsuccessful  offerors  within  five  days  of  a  request. 

3.  Evaluation  Factors  -  The  practice  of  using  contractor  past 
performance  as  an  evaluation  factor.  This  construct  also  includes 
the  practice  of  identifying  in  its  solicitations,  all  significant 
evaluation  factors  and  their  relative  importance. 

Operationalization  of  the  Variables 

There  are  several  strategies  available  to  conduct  research.  Each  strategy  has 
peculiar  advantages  and  disadvantages,  depending  upon  three  conditions:  (a)  the  type  of 
research  question,  (b)  the  control  an  investigator  has  over  actual  behavioral  events,  and 
(c)  the  focus  on  contemporary  as  opposed  to  historical  phenomenon  [Yin,  1994:  1]. 
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Table  1  displays  these  three  conditions  and  shows  how  each  is  related  to  five  major 
research  strategies. 


Table  1.  Relevant  Situations  for  Different  Research  Strategies  [Yin,  1994:  6]. 


Strategy 

Form  of  research  Requires  control 
question  over  behavioral 

events? 

Focuses  on 

contemporary  events? 

experiment 

how, why 

yes 

yes 

survey 

who ,  what , 
where , how 
many, how  much 

no 

yes 

archival 

analysis 

who ,  what , 
where , how 
many, how  much 

no 

yes/no 

history 

how , why 

no 

no 

case  study 

!how,  why 

no 

yes 

To  effectuate  the  objective  of  this  research,  the  author  developed  three 
propositions.  Two  different  strategies  will  be  employed  to  test  the  three  propositions 
since  they  are  based  on  relatively  different  conditions.  The  first  and  third  propositions 
are  related  to  the  impact  of  FASA  on  particular  source  selection  procedures  and  candidate 
proposals.  The  form  of  research  questions  used  in  addressing  these  propositions  entails 
the  use  of  “how”:  (a)  how  did  FASA  impact  Air  Force  source  selection  procedures  as 
they  relate  to  debriefs  and  evaluation  criteria,  and  (b)  how  did  FASA  affect  the  quality  of 
candidate  bids.  Additionally,  addressing  these  two  propositions  would  focus  on 
contemporary  events  and  not  require  control  over  behavioral  events.  Consequently,  a 
case  study  analysis  methodology  is  the  most  appropriate  strategy  to  test  propositions  1 
and  3  according  to  Table  1  [Yin,  1994:  6]. 
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Proposition  2  on  the  other  hand,  could  be  more  appropriately  addressed  through 
an  alternate  strategy  since  it  evokes  a  “what”  form  of  research  question;  what  impact  did 
FASA  have  on  protest  frequency?  Additionally,  addressing  this  proposition  deals  with 
both  contemporary  and  historical  events  and  does  not  require  control  over  behavioral 
events.  Based  on  these  conditions,  an  archival  analysis  is  a  more  suitable  strategy  for 
analyzing  proposition  2  [Yin,  1994:  6]. 

Since  this  study  will  involve  two  different  strategies  of  analysis,  it  will  be  broken 
down  into  two  phases.  As  depicted  in  Figure  1,  the  first  phase  will  be  a  case  study 
method  and  will  consist  of  structured  interviews.  The  second  phase  will  then  be 
conducted  through  archival  analysis  of  protest  data.  Conclusions  will  be  drawn  based  on 
the  combinations  of  findings  from  each  phase.  The  following  sections  discuss  the  two 
strategies  in  more  detail. 

Case  Study.  In  general,  case  studies  are  the  preferred  strategy  when  “how”  or  “why” 
questions  are  being  posed,  when  the  investigator  has  little  control  over  events,  and  when 
the  focus  is  on  a  contemporary  phenomenon  within  some  real-life  context  [Yin,  1994:  1]. 
A  common  misconception  is  that  the  various  research  strategies  should  be  arrayed 
hierarchically.  It  was  once  taught  that  case  studies  were  appropriate  for  the  exploratory 
phase  of  an  investigation,  that  surveys  and  histories  were  appropriate  for  the  descriptive 
phase,  and  that  experiments  were  the  only  way  of  doing  explanatory  or  causal  inquiries 
[Yin,  1994:  3].  The  hierarchical  view  reinforced  the  idea  that  case  studies  were  only  an 
exploratory  tool  and  could  not  be  used  to  describe  or  test  propositions. 
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FASA  had  no 
significant  impact  on 
reducing  bid  protests 


Other  provisions  of  FASA 
dealing  with  areas  other  than 
debriefings  and  evaluation 
criteria  caused  change  or  external] 
factors  were  involved 


The  provisions  of  FASA 
pertaining  to  debriefs  and 
evaluation  criteria  impacted 
protest  frequency 


The  provisions  of  FASA 
pertaining  to  debriefs  and 
evaluation  criteria  did  not 
impact  protest  frequency 


Figure  1.  Design  of  Study. 


This  hierarchical  view,  however  is  incorrect.  Experiments  with  an  exploratory 
motive  have  certainly  always  existed.  In  addition,  the  development  of  causal 
explanations  has  long  been  a  serious  concern  of  historians,  reflected  by  the  subfield 
known  as  historiography  [Yin,  1994:  3].  Finally,  case  studies  are  far  from  being  only  an 
exploratory  strategy.  Some  of  the  best  and  most  famous  case  studies  have  been  both 
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descriptive  (i.e.  Whyte’s  Street  Comer  Society,  1943/1955)  and  explanatory  (Allison’s 
Essence  of  Decision;  Explaining  the  Cuban  Missile  Crisis,  1971)  [Yin,  1994:  3], 

The  more  appropriate  view  of  these  different  strategies  is  a  pluralistic  one.  Each 
strategy  can  be  used  for  all  three  purposes  -  exploratory,  descriptive,  or  explanatory  [Yin, 
1994:  3].  What  distinguishes  the  strategies  is  not  this  hierarchy,  but  the  three  conditions 
discussed  earlier.  As  previously  mentioned,  addressing  the  research  questions  posed  by 
propositions  1  and  3  would  be  most  appropriately  accomplished  through  a  case  study 
strategy  based  on  the  surrounding  conditions. 

The  next  question  is  which  method  to  use  for  data  collection.  Data  collection  for 
case  studies  can  rely  on  many  sources  of  evidence,  including  direct  observation, 
documentation,  physical  artifacts,  and  projective  techniques.  One  of  the  most  important 
sources  of  case  study  information  is  the  interview  [Yin,  1994:  84].  Well  informed 
respondents  can  provide  important  insights  into  a  situation.  They  can  also  provide 
shortcuts  to  the  prior  history  of  the  situation,  helping  to  identify  other  relevant  sources  of 
evidence. 

Therefore,  telephone  interviews  were  conducted  with  PCO’s  throughout  AFMC 
and  its  subordinate  commands  (ASC,  ESC,  SPC)  to  determine  various  impacts  of  FASA 
on  the  systems-level  contracting  process.  The  scope  of  the  interview  is  somewhat  limited 
but  will  provide  some  insight  into  the  impacts  of  the  Act. 

The  goal  of  this  study  will  be  to  provide  insight  into  whether  FASA  reduced 
award  protests  within  AFMC  and  whether  offeror’s  proposals  have  improved  as  a  result 
of  FASA.  AFMC  was  chosen  as  the  unit  of  analysis  in  this  study  because  it  manages 
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approximately  52%  of  the  armual  United  States  Air  Force  budget  and  represents  the 
largest  buying  organization  in  that  service  IWright-Patterson:  Legacy.  1995:  17].  Results 
from  the  interview  should  be  acceptable  for  generalization  to  the  Air  Force  as  a  whole 
since  AFMC  constitutes  such  a  large  proportion  of  the  Air  Force’s  acquisition  activity. 
The  interview  will  concentrate  on  the  systems  arena  versus  operational  contracting  and 
attempt  to  disclose  any  differences  in  particular  areas  of  contracting  policy  to  occur  as  a 
result  of  FASA.  These  areas  will  include  post-award  debriefs,  and  evaluation  factors. 
Additionally,  the  survey  attempted  to  gain  the  interviewees  insight  into  the  impact  of 
these  changes  on  the  frequency  of  award  protests  and  quality  of  proposals.  The  following 
are  the  research  propositions  addressed  by  this  phase  of  the  study: 

Proposition  #1:  FASA  has  not  impacted  the  United  States  Air  Force’s  source 
selection  processes  in  a  manner  that  would  significantly  affect 
post-  award  protest  frequency. 

Proposition  #3:  FASA  has  not  impacted  the  quality  of  proposals  submitted  by 
potential  offerors  in  response  to  Air  Force  solicitations. 

The  interview  was  structured,  to  enhance  reliability,  and  consisted  of  eight 
questions  with  multiple  parts  (see  appendix  A  for  the  interview  script).  The  course  of  the 
interviews  were  determined  by  the  interviewees’  responses  to  the  different  questions. 
Questions  1  through  3  addressed  the  constructs  pertaining  to  “Debrief  Content”  and  were 
intended  to  assess  whether  FASA  increased  the  amount  of  information  disclosed  during 
post-award  debriefs.  If  80%  or  more  of  the  respondents  chose  “a  -  increased  information” 
as  a  response  to  question  1,  the  “Debrief  Content”  construct  reflected  that  “FASA 
resulted  in  the  practice  of  providing  an  increased  amoimt  of  information  regarding 
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reasons  why  unsuccessful  offerors  were  excluded  from  further  consideration  during  post¬ 
award  debriefs.”  The  decision  value  of  80%  was  chosen  because  of  the  desire  to 
conclude  with  a  certain  degree  of  rigor,  considering  the  limited  number  of  interviews 
conducted,  that  particular  responses  indicate  an  impact  in  procedures  has  occurred.  Using 
a  majority  of  60  or  even  70%  would  be  too  conservative  and  would  allow  too  much  room 
for  Type  I  and  Type  II  errors  in  this  methodology.  If  80%  or  more  of  the  respondents 
chose  “a  -  Previously  Done”  to  question  la,  the  “Debrief  Contenf’  construct  reflected  that 
“FAS A  did  not  change  the  amount  of  information  provided  regarding  reasons  why 
unsuccessful  offerors  were  excluded  from  further  consideration  during  post-award 
debriefs.”  If  both  of  the  two  previous  responses  received  less  than  80%,  the  “Debrief 
Content”  construct  reflected  that  “all  of  the  provisions  of  FASA  have  yet  to  be 
implemented  and  their  affects  on  the  amount  of  information  provided  regarding  reasons 
why  unsuccessful  offerors  were  excluded  from  further  consideration  during  post-award 
debriefs  is  unknown.” 

If  80%  or  more  of  the  respondents  chose  “a  -  increased  information”  as  a  response 
to  question  2,  the  “Debrief  Contenf’  construct  reflected  that  “FASA  resulted  in  the 
practice  of  providing  an  increased  amount  of  information  regarding  weaknesses  in 
imsuccessful  offerors’  proposals  during  post-award  debriefs.”  If  80%  or  more  of  the 
respondents  chose  “a  -  procedures  already  adequate”  to  question  2a,  the  “Debrief 
Contenf’  construct  reflected  that  “FASA  did  not  change  the  amotmt  of  information 
provided  regarding  weaknesses  in  unsuccessful  offerors’  proposals  during  post-award 
debriefs.”  If  both  of  the  two  previous  responses  received  less  than  80%,  the  “Debrief 
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Content”  construct  reflected  that  “all  of  the  provisions  of  FASA  have  yet  to  be 
implemented  and  their  affects  on  the  amount  of  information  provided  regarding 
weaknesses  in  unsuccessful  offerors’  proposals  during  post-award  debriefs  is  unknown.” 

If  80%  or  more  of  the  respondents  chose  “a-  included  ranking”  as  a  response  to 
question  3,  the  “Debrief  Content”  construct  reflected  that  “FASA  did  not  change  the 
practice  of  providing  the  overall  ranking  of  all  offerors’  proposals  during  post-award 
debriefs  to  unsuccessful  offerors.”  If  80%  or  more  of  the  respondents  chose  “a  -  now 
includes  ranking”  to  question  3a,  the  “Debrief  Content”  construct  reflected  that  “FASA 
resulted  in  the  practice  of  providing  practice  of  providing  the  overall  ranking  of  all 
offerors’  proposals  to  unsuccessful  offerors  during  post-award  debriefs.”  If  both  of  the 
two  previous  responses  received  less  than  80%,  the  “Debrief  Content”  construct  reflected 
that  “all  of  the  provisions  of  FASA  have  yet  to  be  implemented  and  their  affects  on 
providing  the  overall  ranking  of  all  offerors’  proposals  during  post-award  debriefs  to 
unsuccessful  offerors’  is  unknown.” 

Question  4  addressed  the  “Responsiveness”  construct  and  focused  on  if  FASA 
changed  the  response  time  to  requests  for  debrief.  If  80%  or  more  of  the  respondents 
chose  “a  -  procedures  already  adequate”  as  a  response  to  question  4a,  the 
“Responsiveness”  construct  reflected  that  “FASA  did  not  change  the  response  time  to 
requests  by  unsuccessful  offerors  for  post-award  debriefs.”  If  80%  or  more  of  the 
respondents  chose  “a  -  change  in  procedures”  to  question  4,  the  “Responsiveness” 
construct  reflected  that  “FASA  resulted  in  a  change  in  the  response  time  to  requests  by 
unsuccessful  offerors  for  post-award  debriefs.”  If  both  of  the  two  previous  responses 
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received  less  than  80%,  the  “Responsiveness”  construct  reflected  that  “all  of  the 
provisions  of  FAS  A  have  yet  to  be  implemented  and  their  affects  on  providing  the  overall 
ranking  of  all  offerors’  proposals  during  post-award  debriefs  to  unsuccessful  offerors’  is 
unknown.” 

Question  5  addressed  the  “Evaluation  Factors”  construct  and  was  intended  to 
assess  whether  organizations  started  using  past  performance  as  an  evaluation  factor 
pursuant  to  FASA.  If  80%  or  more  of  the  respondents  chose  “a  -  procedures  already 
adequate”  as  a  response  to  question  5  and  “b  -  procedures  not  changed”  to  question  5c, 
the  “Evaluation  Factors”  construct  reflected  that  “FASA  did  not  change  the  practice  of 
using  past  performance  as  an  evaluation  factor.”  If  80%  or  more  of  the  respondents  chose 
“a  -  procedures  changed”  to  question  5a  or  5c,  the  “Evaluation  Factors”  construct 
reflected  that  “FASA  did  change  the  practice  of  using  past  performance  as  an  evaluation 
factor.”  The  “Evaluation  Factors”  construct  reflected  that  “all  of  the  provisions  of  FASA 
have  yet  to  be  implemented  and  their  affects  on  using  past  performance  as  an  evaluation 
factor  is  unknown,”  in  the  event  that  any  outcome  other  than  those  discussed  above  was 
received. 

Question  6  also  addressed  the  “Evaluation  Factors”  construct  and  was  intended  to 
assess  whether  organizations  started  identifying  in  their  solicitations,  all  significant 
evaluation  factors  and  their  relative  importance  in  the  selection,  pursuant  to  FASA.  If 
80%  or  more  of  the  respondents  chose  “a  -  procedures  already  adequate”  as  a  response  to 
question  6  and  “b  -  procedures  not  changed”  to  question  6c,  the  “Evaluation  Factors” 
construct  reflected  that  “FASA  did  not  change  the  practice  of  identifying  in  solicitations. 
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all  significant  evaluation  factors  and  their  relative  importance  in  the  selection.”  If  80%  or 
more  of  the  respondents  chose  “a  -  procedures  changed”  to  question  6a  or  6c,  the 
“Evaluation  Factors”  construct  reflected  that  “FASA  did  change  the  practice  of 
identifying  in  solicitations,  all  significant  evaluation  factors  and  their  relative  importance 
in  the  selection.”  The  “Evaluation  Factors”  construct  reflected  that  “all  of  the  provisions 
of  FASA  have  yet  to  be  implemented  and  their  affects  on  identifying  in  solicitations,  all 
significant  evaluation  factors  and  their  relative  importance  in  the  selection  is  unknown,” 
in  the  event  that  any  outcome  other  than  those  discussed  above  was  received. 

Question  7  attempted  to  discern  whether  the  contracting  process  has  been  altered 
by  the  Act,  and  if  so,  which  particular  changes  have  had  the  most  influence  on  any 
changes  in  protest  frequency.  It  was  intended  to  be  a  “catch-all”  question  to  identify  any 
areas  that  were  overlooked  by  the  researcher.  Additionally,  it  was  intended  to  gather  the 
practitioners’  opinions  and  insights  into  what  provisions  of  FASA,  if  any,  had  the  greatest 
impact  on  post-award  protest  frequency.  This  data  was  compiled  and  will  be  presented  in 
tabular  format  in  chapters  4  and  5.  If  80%  or  more  of  the  respondents  chose  “a  - 
procedures  changed”  as  a  response  to  question  7,  there  was  a  sufficient  indication  that 
“FASA  did  change  the  organizational  contracting  practices  vsithin  AFMC.”  If  80%  or 
more  of  the  respondents  chose  “b  -  procedures  unchanged”  to  question  7,  there  was  a 
sufficient  indication  that  “FASA  did  not  change  the  organizational  contracting  practices 
within  AFMC.”  Any  other  responses  were  clarified  by  the  results  obtained  in  questions 
7a  or  7b,  which  are  open  ended  questions. 
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Finally,  question  8  was  intended  to  assess  changes  in  the  quality  of  offeror’s 
proposals.  Quality  of  proposals,  for  the  purposes  of  this  study,  is  meant  to  pertain  to  the 
completeness  and  responsiveness  to  all  requirements  of  the  solicitation.  If  80%  or  more 
of  the  respondents  chose  “a  -  improved”  as  a  response  to  question  8  and  “a,  b,  or  c”  to 
question  8a,  there  was  sufficient  indication  that  “FASA  appears  to  have  resulted  in  an 
improvement  in  the  quality  of  candidate  proposals.”  If  80%  or  more  of  the  respondents 
chose  “b  -  degraded”  to  question  8  and  “a,  b,  or  c”  to  question  8b,  there  was  sufficient 
indication  that  “FASA  appears  to  have  resulted  in  a  degradation  in  the  quality  of 
candidate  proposals.”  There  was  sufficient  indication  that  “the  quality  of  candidate 
proposals  does  not  appear  to  have  been  affected  by  the  implementation  of  FASA,”  in  the 
event  that  80%  or  more  of  the  responses  are  from  another  category. 

Table  2  will  serve  as  a  decision  matrix  to  form  conclusions  once  it  is  completed 
by  filling  in  the  response  rates  from  the  interviews.  This  table  will  be  filled  in  with  the 
responses  received  and  used  to  draw  conclusions  in  Chapter  V.  Questions  marked  with 
an  asterisk  are  the  critical  decision  factors  that  will  suggest  a  change  has  occurred  if  at 
least  one  receives  a  response  rate  of  80%  or  greater  and  will  cause  an  overall  rejection  of 
the  null  hypothesis  formed  by  proposition  #1  that  FASA  has  not  impacted  the  United 
States  Air  Force’s  source  selection  processes  in  a  manner  that  would  significantly 
affect  post-award  protest  frequency.  If  none  of  the  questions  marked  with  an  asterisk 
receive  at  least  an  80%  response  rate,  we  fail  to  reject  the  null  hypothesis. 

Additionally,  if  responses  “a  -  improved  proposals”  or  “b  -  degraded  proposals”  to 
question  #8  received  a  response  rate  of  80%  or  greater,  the  null  hypothesis  formed  by 
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proposition  #3  that  FASA  has  not  impacted  the  quality  of  proposals  submitted  by 
potential  offerors  in  response  to  Air  Force  solicitations  was  rejected.  If  response 
“c  -  no  change”  received  at  least  an  80%  response  rate,  we  failed  to  reject  the  null 
hypothesis. 

The  perceptions  of  Air  Force  systems-level  PCO’s  were  sought  for  several 
purposes.  First,  the  contracting  officers  are  at  the  end  of  a  chain  of  events  that  begin  with 
enactment  of  legislation.  Before  a  study  could  be  conducted  to  determine  whether  FASA 
resulted  in  a  change  in  award  protest  frequency,  it  was  first  necessary  to  determine 
whether  FASA  had  any  impact  on  the  way  the  selection  process  is  conducted.  The 
PCO’s,  as  first-  hand  users  of  the  selection  process,  were  in  an  advantageous  position  to 
provide  this  insight.  Special  care  was  taken  to  only  interview  those  PCO’s  who  were 
involved  in  contracting  prior  to  FASA’s  enactment.  Ideally,  this  population  would 
consist  of  those  PCO’s  who  were  involved  in  contracting  prior  to  FASA  and  still  work  for 
the  same  organization.  For  this  reason,  the  population  of  those  interviewed  consisted 
primarily  of  civilian  contracting  officers. 

A  pilot  test  of  the  interview  was  conducted  using  three  Graduate  Contracting 
students  at  the  Air  Force  Institute  of  Technology  (AFIT)  as  respondents.  The  respondents 
were  civilian  contracting  officers  with  significant  experience  in  the  systems-level 
procurement  arena  at  AFMC.  The  test  proved  that  the  questions  were  not  unreasonable 
and  seemed  likely  to  work  in  a  telephone  environment. 
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Table  2.  Decision  Matrix  For  Interview  Responses.  One  or  more  of  any  question  marked 
by  an  asterisk  that  receives  >=  80%  of  the  responses  will  indicate  that  FASA  has  changed 
actual  contracting  practices  within  AFMC. 


QUEST. 


>=  80% 
RESP. 


CONCLUSION 


DECISION 


1,2 


FASA  resulted  in  the  practice  of  providing  an  increased  amount  of 
information  to  unsuccessful  offerors  during  post-award  debriefs 


Refer  to  question  la 


Insufficient  evidence  exists  to  make  determination 


la, 2a 


FASA  did  not  result  in  the  practice  of  providing  an  increased 
amount  of  information  to  unsuccessful  offerors  during  post-award 
debriefs 


B,C,D 


Insufficient  evidence  exists  to  make  determination 


FASA  did  not  result  in  the  practice  of  providing  an  increased 
amount  of  information  to  unsuccessful  offerors  during  post-award 
debriefs 


B 


Refer  to  question  3a 


Insufficient  evidence  exists  to  make  determination 


3a 


FASA  resulted  in  the  practice  of  providing  an  increased  amount  of 
information  to  unsuccessful  offerors  during  post-award  debriefs 


B,C 


Insufficient  evidence  exists  to  make  determination 


FASA  changed  the  response  time  to  requests  by  unsuccessful 
offerors  for  post-award  debriefs 


B 


Refer  to  question  3a 


Insufficient  evidence  exists  to  make  determination 


4a 


FASA  did  not  change  the  response  time  to  requests  by 
unsuccessful  offerors  for  post-award  debriefs 


B.C.D 


Insufficient  evidence  exists  to  make  determination 


5a, 6a 


FASA  changed  evaluation  factor  practices 


B 


FASA  has  not  changed  evaluation  factor  practices 


Insufficient  evidence  exists  to  make  determination 


5c,6c 


FASA  changed  evaluation  factor  practices 


FASA  did  not  change  evaluation  factor  practices 


Insufficient  evidence  exists  to  make  determination 


FASA  changed  contracting  practices 


B 


FASA  has  not  changed  contracting  practices 


Insufficient  evidence  exists  to  make  determination 
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Additionally,  the  interview  checklist  was  reviewed  for  content  and  validity  by 
Major  Caisson  Vickery,  Ph.D.  Major  Vickery  is  a  contracting  officer  with  significant 
experience  in  both  operational  and  systems  contracting  and  is  currently  an  Assistant 
Professor  of  Contract  Management  at  AFIT’s  Graduate  Contracting  Management 
Program.  His  contributions  aided  the  author  in  further  refining  the  reliability  of  the 
questionnaire. 

The  scope  of  the  interviews  included  five  interviews  at  ASC,  five  interviews  at 
ESC,  and  five  interviews  at  SPC,  for  a  total  of  15  interviews.  Reductions  in  personnel 
and  large  Government  procurements  has  diminished  the  available  pool  of  potential 
subjects  with  the  requisite  experience  for  the  interview.  Therefor  the  number  of 
interviews  established  for  this  study  was  based  on  a  desire  to  obtain  an  adequate  amount 
and  spread  of  data  with  the  limited  available  resources.  The  individuals  interviewed  were 
chosen  with  the  assistance  of  highly  placed  military  or  civilian  contacts.  For  specifics  on 
the  interview,  refer  to  the  script  found  in  Appendix  A. 

As  indicated  by  Figure  1,  this  was  a  two-part  study  and  it  was  necessary  to  gather 
two  types  of  information  in  order  to  draw  any  conclusions.  The  first  type  of  information, 
whether  FAS  A  has  changed  contracting  practices  within  AFMC,  was  previously 
discussed.  Once  this  information  was  obtained,  the  next  step  was  to  determine  whether 
there  has  been  a  change  in  the  frequency  of  actual  bid  protests  occurring  annually.  The 
next  section  will  discuss  what  methods  used  in  this  analysis. 

Protest  Frequency  Analysis.  Since  protest  frequency  is  indicative  of  contractor 
willingness  to  file  protests,  it  would  be  significant  if  trends  differ  before  and  after  FASA. 
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Data  was  collected  on  the  number  of  bid  protests  filed  against  ASC  annually,  both  before 
the  enactment  of  FASA  and  afterwards.  These  periods  included  data  from  fiscal  years 
1993  through  1996  with  preliminary  data  through  June  of  fiscal  year  1997.  The  data 
consisted  of  total  protests  received  and  number  of  new  competitive  contract  awards  of  $3 
million  or  more. 

The  data  was  limited  as  previously  mentioned,  to  ASC  since  this  type  of  data  is 
not  tracked  by  the  other  two  sub-commands.  This  limitation  should  not  impact  the 
external  validity  of  the  study  however.  The  representativeness  of  a  sample  refers  to  “the 
degree  of  similarity  between  the  characteristics  of  the  sample  and  the  characteristics  of 
the  universe  from  which  the  sample  is  drawn,”  [Clover  and  Balsley,  1984:  81].  “Ideally, 
the  characteristics  with  which  the  study  is  concerned  should  be  found  in  the  same 
proportions  as  they  occur  in  the  universe,”  [Clover  and  Balsley,  1984:  81].  In  this  phase 
of  the  study,  the  universe  is  considered  to  be  protest  frequency  within  AFMC.  The 
sample  is  the  protest  frequency  within  ASC.  The  following  reasons  are  given  for 
justification  of  the  representativness  of  ASC  data:  (1)  the  activities  of  all  three  sub¬ 
commands  center  around  large,  major-system  type  procurements  for  the  Air  Force;  and 
(2)  “Lightning  Bolt”  #4  mandated  the  removal  of  center-level  policies,  therefor  all  three 
sub-commands  should  be  operating  under  the  same  acquisition  regulations.  Various 
methods  are  available  to  attempt  to  determine  whether  trends  and  any  shift  corresponding 
to  FASA  are  identifiable. 

The  first  analysis  consisted  of  a  year  by  year  comparison  of  the  protests  to  discern 
whether  there  has  been  any  changes  over  the  5  year  period.  Proper  consideration  was 


45 


given  to  the  reduction  of  the  number  of  procurements  and  the  effects  of  other  Federal 
Acquisition  Reform  initiatives.  This  was  partially  accomplished  by  using  protests  as  a 
percentage  of  total  contracts  awarded  that  same  year.  Each  year’s  data  consisted  of  the 
number  of  protests  as  a  proportion  of  contract  awards  of  $3  million  or  greater  that  same 
fiscal  year  (p;).  The  research  hypotheses  for  this  test  are  as  follows: 

Ho;  Pl=P2=Pi=P4=P5 
H,:  Pi*  Pi^  Pi^Pi*  Ps 

The  second  analysis  consisted  of  a  comparison  between  the  pre-  and  post-FASA 

periods  to  discern  whether  there  is  a  difference  between  the  two  periods.  The  pre-FASA 

period  consisted  of  fiscal  years  1993-1994,  and  the  post-FASA  period  consisted  of  1996- 

1997.  Fiscal  year  1995  was  considered  to  be  a  transition  period  since  the  vmious 

provisions  of  the  Act  were  implemented  at  different  times  with  overall  implementation 

completed  by  October  1,  1995.  The  research  hypotheses  for  this  test  are  as  follows: 

H^:  Pi  =  P2 
H,:  Pi  *  P2 

There  were  several  alternatives  available  to  conduct  an  analysis  of  these  data. 
Ideally  we  would  have  wanted  to  use  a  parametric  statistical  test  since  they  make 
maximum  use  of  all  the  information  that  is  inherent  in  the  data  [Runyon  and  Haber,  1971 
236].  Additionally,  parametric  tests  were  preferred  for  this  analysis  because  they  are 
more  powerful  and  contain  less  risk  of  failing  to  reject  a  false  null  hypothesis  [Runyon 
and  Haber,  1971 :  235].  In  order  to  use  such  tests,  however  it  was  first  necessary  to 
ensure  that  the  data  lent  themselves  to  analysis  in  terms  of  the  normal  probability  curve, 
or  met  the  basic  assumptions  for  their  application  [Runyon  and  Haber,  1971 :  229].  The 


46 


first  assumption  required  that  the  samples  be  drawn  from  normal  populations.  This 
particular  assumption  is  where  we  encountered  a  problem  with  this  data.  The  population, 
which  consisted  of  each  fiscal  years  protests  within  the  Air  Force,  from  which  the 
samples  were  drawn,  are  not  of  sufficient  size  to  assume  normality  under  the  Central 
Limit  Theorem.  Additionally,  the  data  is  categorical,  either  there  were  protests  or  not, 
and  this  precluded  the  assumption  of  normalcy  as  well. 

When  the  assumptions  imderlying  a  statistical  test  are  violated,  the  test  may  lose 
much  of  its  strength  [VanMatre  and  Gilbreath,  1980;  466].  In  the  instance  where  the  data 
do  not  lend  themselves  to  these  prerequisites,  it  may  become  necessary  to  use  a  non- 
parametric  test.  Since  the  data  could  not  support  the  assumptions  required  for  a 
parametric  method,  it  was  necessary  to  use  the  non-parametric  Chi-square  for  this  study. 

The  Chi-square  test  for  independence  appeared  to  be  the  most  appropriate  since 
there  were  two  or  more  related  or  matched  samples  and  our  objective  was  to  determine 
whether  the  samples  are  drawn  from  identical  populations  [Ott,  1988:  253].  The 
following  assumptions  for  the  Chi-square  test  are  quite  unrestrictive  and  do  not  specify 
the  form  and/or  shape  of  the  population  form  which  the  sample  is  drawn:  (l)the  expected 
frequency  for  each  cell  should  be  at  least  five;  (2)  the  scale  of  measurement  is  at  least 
nominal;  (1)  the  samples  are  random  and  independent;  and  (4)  the  data  in  each  sample  is 
frequency  in  form  and  categorized  into  mutually  exclusive  and  exhaustive  categories 
[Brewer,  1991:  7-14].  The  assumptions  of  the  Chi-square  test  was  satisfied  by  the  data 
and  the  determination  of  the  data’s  suitability  for  this  test  is  discussed  in  Chapter  IV. 
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All  of  the  hypotheses  were  tested  to  the  .05  level  of  significance  to  minimize  risk 
of  Type  I  error.  The  results  from  the  second  analysis  comparing  the  two  periods  was 
used  to  address  the  second  proposition  posed  in  chapter  II,  FASA  has  not  impacted 
post-award  protest  frequency  within  the  United  States  Air  Force.  Failing  to  reject 
the  null  hypothesis  at  the  .05  level  of  significance  would  indicate  that  the  samples  are 
drawn  from  identical  populations  and  that  FASA  has  not  impacted  protest  frequency 
within  the  Air  Force.  The  results  from  the  first  test  between  each  individual  fiscal  year 
was  used  to  confirm  those  results  and  provide  further  analysis  for  discussion. 

Summary 

In  order  to  successfully  test  the  propositions  of  this  study,  it  was  necessary  to 
conduct  a  two  phase  approach.  The  first  phase  consisted  of  structured  telephone 
interviews  and  was  intended  to  yield  data  that  answered  the  question  as  to  whether 
particular  practices  have  changed  as  a  result  of  FASA.  These  interviews  also  provided 
insight  into  whether  contractor  proposals  have  improved  in  quality  as  a  result  of  FASA. 

The  second  phase  employed  statistical  analysis  to  discern  whether  the  frequency 
of  post-award  protests  has  changed  over  a  relevant  time  period.  The  results  from  both 
phases  of  the  research  was  combined  to  arrive  at  our  conclusions,  as  indicated  by  the 
study  design  in  Figure  1 .  The  next  chapter  provides  the  results  of  the  analyses  that  was 
discussed  above. 
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IV.  Analysis 


General 

This  chapter  presents  the  data  collected  and  analysis  of  the  answers  provided  by 
the  respondents  to  the  structured  interviews.  The  first  section  will  detail  the  results  of  the 
structured  interviews.  Responses  will  be  broken  down  demographically  by  command  and 
function  to  identify  any  anomalies  or  trends  that  may  occur.  The  next  section  will  present 
the  protest  data  collected  from  various  agencies.  Additionally,  a  discussion  of  the  test 
selection  based  on  the  properties  of  the  data  will  be  included  along  with  results  of  the 
statistical  analysis  that  was  conducted. 

Structured  Interview  Results 

Fifteen  structured  telephone  interviews  were  conducted  with  PCO’s  throughout 
AFMC  and  its  subordinate  commands  (ASC,  ESC,  SPC)  to  determine  various  impacts  of 
FASA  on  the  systems-level  contracting  process.  The  number  of  interviews  were  divided 
evenly  between  the  commands  with  each  receiving  five  interviews. 

As  previously  mentioned  in  Chapter  III,  PCO’s  were  chosen  since  they  are  first¬ 
hand  users  of  the  source  selection  process  and  could  provide  valuable  insights  into  any 
changes  that  might  have  occurred.  Potential  candidates  for  the  interviews  were  asked 
screening  questions  to  ensure  that  they  had  adequate  source  selection  experience  both 
pre-  and  post-FASA.  The  necessary  experience  requirements  often  resulted  in  active  duty 
military  officers  being  excluded  as  candidates  for  the  interview  since  they  tend  to  rotate 
positions  more  frequently  than  their  civilian  counterparts.  Consequently,  all  but  one  of 
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the  interviewees  were  civil  service  employees.  The  author  also  attempted  to  equitably 
distribute  interviews  between  PCO’s  in  system  program  offices  (SPO’s),  and  support 
agencies,  such  as  source  selection  agencies  and  policy  offices  within  the  three  subordinate 
commands.  This  proved  to  be  rather  challenging  however  due  to  reductions  in  defense 
spending  and  acquisition  personnel,  fewer  new  programs,  and  a  preponderance  of  sole 
source  contracts.  Despite  the  difficulties  the  author  was  able  to  maintain  a  60%  to  40% 
split,  with  PCO’s  in  system  program  offices  being  in  the  minority.  The  following  section 
will  discuss  the  responses  received. 

Findings.  In  this  section,  responses  to  each  question  are  presented  and  analyzed  by 
subordinate  command  and  personnel  demographics  (SPO  versus  support  personnel). 
Tables  8-12  found  in  Appendix  B  provide  a  tabular  representation  of  the  responses  and 
break  them  down  demographically  by  command,  and  support  versus  SPO  personnel.  The 
contents  of  these  tables  will  be  discussed  in  further  detail,  question  by  question. 

Question  #1 :  Pursuant  to  FAS  A,  has  your  organization  increased  the 

amount  of  information  provided  to  unsuccessful  offerors  during 
post-award  debriefs  regarding  reasons  why  they  were  excluded 
from  further  consideration? 

a)  Yes[93%] 

b)  No[7%] 

c)  Don’t  know  [0] 

Analysis:  Table  8  contains  the  overall  responses  to  the  interview  throughout  AFMC. 
The  first  number  in  each  cell  is  the  total  number  of  interviewees  that  chose  that  particular 
response.  The  second  number  is  the  percentage  of  interviewees  that  chose  that  response. 
The  interview  was  structured  in  a  manner  that  different  responses  may  have  dictated 
additional  or  different  questions  be  asked  for  clarification  or  additional  detail.  That 
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explains  why  some  rows  add  up  to  less  than  the  total  number  of  respondents.  Some  of 
the  questions  that  were  asked  were  open  ended  questions  and  are  not  included  in  any  of 
the  tables.  The  responses  received  to  those  questions  are  discussed  throughout  this 
section.  Refer  to  the  Appendix  for  a  copy  of  the  structured  interview  script. 

Referring  to  Table  8,  93  %  of  those  interviewed  responded  “yes”  to  question  1, 
while  only  7%  responded  “no.”  Tables  9, 10,  and  1 1  break  the  responses  down  by 
subcommand.  As  previously  mentioned,  the  scope  of  this  study  was  limited  to  AFMC 
and  the  interviews  were  conducted  equitably  between  its  three  primary  subcommands, 
ASC,  ESC,  and  SMC.  Tables  9, 10,  and  1 1  in  Appendix  B  have  the  same  format  as  Table 
8,  with  the  first  number  representing  the  total  number  of  each  response  received  and  the 
percentage  of  interviewees  that  responded  with  that  option. 

Table  12  depicts  the  responses  demographically  by  support  agency  versus  system 
program  office.  The  format  of  Table  12  varies  slightly  from  the  previous  tables.  Each 
response  cell  is  separated  into  two  categories:  support  and  SPO.  The  numbers  in  each 
cell  represent  the  percentage  of  those  interviewees  in  each  category  that  responded  with 
that  particular  option.  As  was  the  case  with  the  previous  tables,  interviewees  may  have 
been  asked  a  different  sequence  of  questions  based  on  previous  responses.  Consequently, 
while  the  responses  to  each  question  may  add  up  to  100%,  it  is  based  on  those  required  to 
answer  that  particular  question,  and  not  the  entire  group  as  a  whole. 

Additionally,  the  following  charts  (Figure  2)  provide  a  graphic  representation  of 
the  responses.  These  charts  are  a  summation  of  that  data  presented  in  Tables  8  through 
12  and  pertains  to  question  1. 
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Response  by  Sub-Command  Re^onse  by  Demographics 
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Figure  2.  Graphic  Response  Representation  For  Question  #1 . 

All  of  those  surveyed,  with  the  exception  of  one,  felt  that  their  organization 
increased  the  amount  of  information  provided  to  unsuccessful  offerors  during  post-award 
debriefs  regarding  reasons  why  they  were  excluded  from  further  consideration  pursuant  to 
FASA.  The  lone  respondent  that  chose  answer  “b”  for  question  1  felt  that  their 
organization  was  already  providing  informational  content  during  debriefs  that  met  or 
surpassed  FASA  requirements.  Therefor,  he  felt  that  FASA  had  no  impact  on  his 
organization’s  contracting  process  in  this  particular  area. 

Question  #2:  Pursuant  to  FASA,  has  your  organization  increased  the 

amount  of  information  provided  to  unsuccessful  offerors  during 
post-award  debriefs  regarding  weaknesses  in  their  proposal? 

a)  Yes[93%] 

b)  No[7%] 

c)  Don’t  know  [0] 

Analysis:  Question  #2  was  very  similar  to  question  #1  but  was  intended  to  gather 
additional  insights  into  whether  FASA  impacted  debrief  content.  The  responses  to  this 
question  were  identical  to  those  encountered  in  question  #1,  with  all  respondents,  with  the 


exception  of  one,  maintaining  that  there  has  been  an  increase  in  information  provided  to 


unsuccessful  offerors  during  post-award  debriefs.  The  same  individual  who  answered 
“no”  to  question  1  also  answered  “no”  to  question  #2.  He  used  the  same  reasoning  that 
their  organization  was  already  providing  informational  content  during  debriefs  that  met  or 
surpassed  FAS  A  requirements.  Therefor,  he  felt  that  FAS  A  had  no  impact  on  his 
organization’s  contracting  process  in  this  particular  area  as  well. 

Question  #3;  Did  your  organization  include  the  overall  ranking  of  all 

offerors  during  post-award  debriefs  to  unsuccessful  offerors 
prior  to  FASA? 

a)  Yes  [0] 

b)  No  [93%] 

c)  Don’t  know  [7%] 

Analysis:  Once  again,  there  was  nearly  a  complete  agreement  on  the  answer  to  this 
question.  All  of  those  interviewed,  with  the  exception  of  one,  agreed  that  their 
organizations  did  not  provide  some  form  of  ranking  during  post-award  debriefs  prior  to 
FASA.  Tables  8  through  12  break  down  the  responses  by  sub-command  and  limited 
demographics  and  the  Figure  3  provides  a  graphic  representation  of  the  responses. 
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Figure  3.  Graphic  Response  Representation  For  Question  #3. 
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As  indicated,  there  was  once  again  a  lone  dissident  in  the  group  of  respondents. 
This  time  however,  the  respondent  was  not  in  disagreement  with  the  others  but  was 
unsure  of  the  answer,  so  he  answered  “don’t  know.”  This  individual  was  not  the  same 
subject  that  responded  differently  than  the  others  in  questions  1  and  2.  In  order  to 
determine  whether  FASA  had  any  impact  on  the  organization-level  contracting  process  in 
this  area,  the  next  question  focused  on  whether  any  changes  had  occurred  in  this  arena 
after  FASA’s  implementation. 

Question  #3a:  Does  your  organization  now  include  the  overall  ranking  of 
all  offerors  during  post-award  debriefs  to  unsuccessful 
offerors? 

a)  Yes  [33%] 

b)  No  [60%] 

c)  Don’t  know  [7%] 

Analysis;  This  question  became  one  of  the  early  discriminators  in  the  interview. 

Most  of  the  respondents  agreed  in  the  previous  question  that  their  organizations  did  not 
provide  some  form  of  ranking  during  debriefs.  However,  question  #3a  demonstrated  that 
there  was  less  agreement  as  to  whether  this  practice  was  implemented  after  FASA’s 
introduction,  with  60%  saying  that  FASA  did  not  affect  practices  in  this  area  and  33% 
saying  that  they  were.  Tables  8  through  12  break  down  the  responses  by  sub-command 
and  limited  demographics  and  Figure  4  provides  a  graphic  representation  of  the 
responses. 
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Figure  4.  Graphic  Response  Representation  For  Question  #3a. 


The  respondent  that  chose  “don’t  know”  in  question  3  chose  the  same  response 
for  questions  3a  and  3b  as  well.  Figure  3  above  illustrates  that  a  majority  of  ASC 
personnel  tended  to  believe  that  FASA  had  changed  their  contracting  practices  in  this 
area.  The  other  two  commands  shared  their  disagreement  however  and  seemed  to 
indicate  that  nothing  has  changed  in  this  area.  Interestingly  enough  however  was  that 
there  was  virtually  no  disagreement  between  SPO  and  support  personnel. 

A  majority  (89%)  of  those  that  responded  that  there  had  been  no  such  change  as  a 
result  of  FASA  indicated  that  such  provisions  were  not  applicable  to  their  organizations 
for  one  reason  or  another.  Additionally,  they  agreed  (89%)  that  they  did  not  expect  such 
provisions  to  be  implemented  at  their  organizations  any  time  in  the  near  future. 

Question  #4:  Pursuant  to  FASA,  has  your  organization  changed  its 
response  time  to  debrief  requests  made  by  unsuccessful 
offerors? 

a)  Yes  [67%] 

b) No  [33%] 

c)  Don’t  know  [0] 
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Analysis;  This  question  is  another  instance  where  the  results  may  not  be 
unquestionably  conclusive  despite  the  fact  that  the  spread  is  greater  than  the  responses 
from  question  3  and  none  of  the  subjects  responded  with  “don’t  know.”  Tables  8  through 
12  break  down  the  responses  by  sub-command  and  limited  demographics  and  Figure  5 
provides  a  graphic  representation  of  the  responses. 
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Figure  5.  Graphic  Response  Representation  For  Question  #4. 


The  responses  vary'  slightly  betw'een  ASC  and  the  other  sub-commands,  how'ever 
they  still  somewhat  close.  Respondents  in  support  agencies  indicated  that  their 
organizations  have  decreased  their  response  times  as  a  result  of  FAS  A  however  the 
SPO’s  were  evenly  split  on  the  issue.  Those  that  indicated  that  response  times  have  not 
changed  were  in  complete  agreement  that  pre-existing  procedures  and  practices  either  met 
or  exceeded  the  timelines  called  for  by  the  provisions  of  FASA. 
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Question  #5;  Did  your  organization  include  contractor  past 
performance  as  an  evaluation  factor  prior  to  the 
enactment  of  FAS  A? 

a)  Yes  [80%] 

b) No  [20%] 

c)  Don't  know  [0] 


Analysis;  The  overall  results  from  question  5  seem  to  be  more  conclusive  than  some 
of  the  previous  questions.  A  majority  of  those  interviewed  maintained  that  contractor 
past  performance  was  used  as  an  evaluation  factor  prior  to  FASA.  However,  as  indicated 
by  Figure  6,  there  was  some  disparity  in  responses  between  the  subcommands. 
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Figure  6.  Graphic  Response  Representation  For  Question  #5. 


While  there  was  little  differences  in  responses  between  SPO  and  support 
personnel,  there  was  a  notable  dissimilarity  in  responses  between  the  sub-commands. 
ASC  personnel  unanimously  responded  that  past  performance  was  used  prior  to  FASA 
while  ESC  was  nearly  evenly  split.  Those  that  responded  that  their  organization  had  not 
used  contractor  past  performance  prior  to  FASA  were  further  questioned  whether  their 
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organizations  now  use  that  data  pursuant  to  FAS  A.  Sixty-seven  percent  of  those 
respondents  indicated  that  their  organizations  now  use  past  performance.  There  was  quite 
a  disparity  between  the  sub-commands  in  this  particular  area.  ESC  personnel  were  evenly 
split  as  to  whether  any  change  in  this  area  has  occurred,  while  SMC  responded 
unanimously  that  their  organizations  now  use  this  data,  indicating  FASA  had  impacted 
this  area  of  their  contracting  process.  As  previously  mentioned,  ASC  personnel 
unanimously  responded  that  past  performance  was  used  prior  to  FASA  so  they  were  not 
administered  this  particular  question.  These  results  seem  to  indicate  that  the  three 
different  sub-commands  may  have  been  using  three  distinctly  different  practices  in  the 
area  of  evaluation  factors. 

Those  subjects  that  indicated  that  past  performance  data  was  used  by  their 
organizations  as  evaluation  factors  prior  to  FASA  were  further  questioned  as  to  whether 
FASA  may  have  changed  the  way  that  this  data  was  used  in  source  selections  after  the 
Act’s  implementation. 

Question  5c:  Did  the  implementation  of  FASA  change  the  way  your 
organization  used  contractor  past  performance  as  an 
evaluation  factor 

a)  Yes  [69%] 

b) No  [31%] 

c)  Don’t  know  [0] 

Responses  varied  once  again,  both  between  sub-commands  and  across  the 
demographic  variables  as  indicated  in  Figure  7. 
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Figure  7.  Graphic  Response  Representation  For  Question  #5c. 


ESC  seemed  to  rather  strongly  disagree  with  the  other  two  commands  in  this  area. 
This  could  be  attributed  to  a  few  factors.  As  previously  indicated,  ESC  was  nearly  evenly 
split  as  to  whether  their  organization  used  past  performance  data  as  evaluation  factors 
prior  to  FASA  while  the  other  two  commands  tended  to  feel  more  strongly  that  their 
organizations  did  use  past  performance  data.  Since  the  number  of  respondents  at  ESC 
who  felt  this  way  w'as  smaller  than  the  other  groups,  their  overall  responses  were  more 
sensitive  to  slight  changes.  Additionally,  there  was  a  fairly  large  disparity  between  the 
two.  SMC  interviewees  consisted  of  the  largest  percentage  of  support  versus  SPO 
personnel  in  the  three  groups  and  could  have  contributed  to  this  sizable  difference.  As 
previously  mentioned,  maintaining  an  even  representation  between  the  three  sub¬ 
commands  was  difficult  due  to  the  changing  defense  procurement  environment.  This  was 
particularly  true  at  SMC. 
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Question  #6:  Did  your  organization  identify  in  its  solicitations,  all 
significant  evaluation  factors  and  their  relative 
importance  prior  to  the  enactment  of  FAS  A? 

a)  Yes  [93%] 

b) No  [7%] 

c)  Don’t  know  [0] 

Analysis:  Responses  to  this  question  appeared  to  be  much  more  conclusive  that  some 
of  the  previous  questions.  All  subjects  with  the  exception  of  one  indicated  that  their 
organizations  had  identified  all  significant  evaluation  factors  an  their  relative  importance 
prior  to  FASA’s  implementation.  The  one  interviewee  that  responded  otherwise  could  be 
grouped  under  the  support  personnel  demographic  variable  at  SMC.  That  respondent  did 
indicate  through  further  questioning  that  his  organization  does  presently  utilize  that 
practice  as  a  result  of  FAS  A,  however.  Those  that  indicate  that  their  organizations  did 
include  this  information  in  solicitations  were  further  queried  as  to  whether  FASA 
changed  the  way  that  they  identified  this  information. 

Question  #6c:  Did  the  implementation  of  FASA  change  the  way 
your  organization  identified  in  its  solicitations,  all 
significant  evaluation  factors  and  their  relative 
importance? 

a)  Yes  [57%] 

b) No  [36%] 

c)  Don’t  know  [7%] 

Analysis:  Overall,  the  results  from  this  question  do  not  appear  to  be  very  conclusive. 
However,  if  you  break  down  the  results  by  sub-command  they  do  appear  somewhat  more 
so.  As  has  previously  been  the  case  in  this  research  though,  there  is  a  disparity  in 
responses  between  ESC  and  the  other  two  sub-commands.  Refer  to  Figure  8. 
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Figure  8.  Graphic  Response  Representation  For  Question  #6c. 


One  respondent  from  a  support  organization  at  ESC  was  not  certain  whether  there 
were  any  changes  in  this  areas  to  occur  pursuant  to  FASA,  so  he  chose  “don’t  know.” 
The  next  question  was  meant  to  get  the  subjects’  overall  opinion  as  to  whether  FASA 
changed  their  organization’s  contracting  process. 

Question  #7:  Overall,  would  you  say  that  FASA  changed  your 
organization’s  contracting  process? 

a)  Yes  [93%] 

b) No  [7%3 

c)  Don’t  know  [0] 

Analysis:  It  was  interesting  to  note  that  a  few  of  the  interviewees  indicated  that  the 
changes  to  occur  as  a  result  of  F.4S  A  were  quite  significant  while  a  majority  of  the 
subjects  thought  the  changes  to  be  on  a  smaller  scale.  Regardless  of  the  scale,  all  of  the 
respondents,  with  the  exception  of  one  from  a  support  organization  at  ESC,  maintained 
that  FASA  did  impact  their  organization’s  contracting  process  in  one  way  or  another. 

The  respondent  that  answ^ered  “no”  to  this  question  indicated  that  the  implementation  of 
F.ASA  was  a  positive  step  towards  improving  federal  acquisitions,  however  her 
organization’s  practices  were  consistent  with  the  Act’s  provisions  prior  to  its  creation. 
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She  expounded  further  by  saying  that  she  believed  that  this  was  the  case  throughout  the 
DOD  and  that  FASA’s  impact  would  be  greatest  in  other  executive  agencies  outside  the 
DOD.  The  responses  of  the  other  interviewees  seemed  to  contradict  her  notions  however. 

Those  respondents  that  indicated  that  changes  had  occurred  were  further 
questioned  about  what  they  thought  to  be  the  most  significant  change  to  occur  as  a  result 
of  FAS  A.  The  responses  included  the  following: 

•  increase  in  debrief  content  (openness)  -  [57%] 

•  increased  thresholds  -  [14%] 

•  expedition  of  debrief  (timing)  -  [7%] 

•  inducement  to  “commercial  off  the  shelf’  -  [7%] 

•  changes  in  organizational  structure  -  [7%] 

•  early  industry  involvement  -  [7%] 

The  interviewees  were  then  questioned  as  to  whether  they  thought  that  there  has 
been  any  change  in  the  number  of  protests  occurring  since  FASA’s  implementation. 

Question  #7c:  Overall,  would  you  say  that  award  protests  have 

increased  or  decreased  as  a  result  of  these  changes 

made  in  your  organizations  contracting  process? 

a)  Increased  [14%] 

b)  Decreased  [50%] 

c)  Neither  [29%] 

d)  Don’t  Know  [7%] 

Analysis:  While  most  of  the  respondents  agreed  that  FASA  impacted  their 
organization’s  contracting  process,  there  was  little  consensus  on  the  effect  of  these 
changes.  Figure  9  breaks  down  the  responses  by  demographic  variables  and  sub¬ 
command  locations. 


62 


Figure  9.  Graphic  Response  Representation  For  Question  #7c. 


As  illustrated  by  Figure  9,  there  does  not  appear  to  be  much  of  a  consensus  at  any 
level  of  this  analysis.  The  only  similarity  between  the  demographic  responses  appears  to 
be  that  both  groups  agree  that  protests  have  not  increased  as  a  result  of  FAS  A.  ESC 
personnel  seem  to  be  evenly  split  as  to  w'hether  protests  have  increased.  Those  that 
believed  that  protests  have  increased  offered  use  of  past  performance  as  an  evaluation 
factor  and  desperate  contractors  w^anting  "their  piece  of  the  shrinking  defense 
procurement  pie”  as  the  most  common  reasons  for  the  increases  at  50%  each.  Those  that 
believed  protests  have  decreased,  unanimously  agreed  that  the  increase  in  information 
provided  during  debriefs  was  the  primary'  factor  in  the  reductions.  Given  this  consensus, 
the  next  question  was  intended  to  get  PCOs'  opinions  on  whether  this  increase  in 
information  might  have  resulted  in  an  improvement  in  the  quality  of  contractor  proposals. 

Question  #8:  Do  you  feel  that  the  quality  of  proposals  submitted  have 

changed  in  regards  to  completeness  and  responsiveness  to  the 
solicitation  since  the  implementation  of  FAS.A.? 

a)  Improved  [33%] 

b)  Degraded  [7%] 

c)  No  change  [60%] 

d)  Don’t  know  [0] 
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Analysis:  The  responses  to  this  question  were  surprising  to  the  author.  Common 
sense  seems  to  imply  that  if  contractors  are  getting  more  information  about  their 
weaknesses  during  debriefs  that  they  might  use  this  information  to  create  better  proposals 
in  the  future.  However,  the  responses  to  this  question  seem  to  indicate  otherwise.  While 
a  small  percentage  feel  that  proposals  have  improved,  a  slight  majority  feel  that  there  has 
been  no  change. 

Figure  10  illustrates  that  most  respondents  in  ESC  and  ASC  felt  that  there  has 
been  no  change  in  the  quality  of  proposals,  however  SMC  was  somewhat  undecided  with 
"improvement”  receiving  the  largest  representation  at  40%.  “No  change”  also  received  a 
slight  majority  across  both  demographic  variables.  Those  that  did  indicate  that  their  has 
been  an  improvement  responded  unanimously  that  the  primary  contributing  factor  to  this 
phenomenon  has  been  the  increase  in  information  provided  at  debriefs.  The  sole 
interviewee  that  felt  that  the  quality  of  proposals  have  degraded  attributed  this 
phenomenon  to  Government  and  contractor  attitudes.  He  said  that  the  Government  is  too 
accommodating  in  certain  areas  and  contractors  tend  to  rely  on  this  generosity. 


Figure  10.  Graphic  Response  Representation  For  Question  #8. 
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The  next  section  discusses  the  analysis  of  the  archival  data  that  constitutes  the 
second  phase  of  this  study. 

Archival  Analysis 

As  discussed  in  chapter  III,  the  second  phase  of  this  study  consists  of  a  statistical 
analysis  of  protest  data  by  fiscal  year  within  ASC.  The  protest  data  consists  of  the 
number  of  protests  received  within  ASC  each  fiscal  year  as  a  percentage  of  the  total 
contract  awards  of  $3  million  or  greater  for  the  same  fiscal  year.  The  number  of  contract 
awards  meeting  the  above  criteria  for  each  year  was  gathered  from  AFMC’s  Automated 
Management  Information  System  (AMIS)  office  at  Wright-Patterson  Air  Force  Base  in 
Ohio.  The  number  of  protests  received  on  those  awards  was  obtained  through  the  Air 
Force  Contract  Law  Center,  which  is  also  located  at  Wright-Patterson  Air  Force  Base. 

The  following  table  depicts  the  data  that  was  obtained  and  will  be  used  as  the  focus  of  the 
of  the  second  phase  of  this  study. 


Table  3.  ASC  Protest  Data. 


hiscal  Year  | 

mm 

ggyj] 

— S7^ 

#  of  Competitive  Awards  >=  $3M 

mm 

■1^ 

#  of  Protests 

mm 

mm 

imgg 

Protests  as  Percentage  ot  Awards 

ISBSSI 

IS^ 

igyngj 

As  mentioned  in  chapter  III,  the  preferred  method  to  use  in  analyzing  this  data 
was  a  parametric  statistical  method.  However,  the  data  being  used  could  not  satisfy  the 
assumptions  of  such  a  test. 
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The  Chi-square  test  has  assumptions  to  meet  as  well  although  they  are  much  less 


stringent  than  those  required  for  its  parametric  counterparts.  The  first  assumption 


requires  that  the  expected  frequency  for  each  cell  should  be  at  least  five.  The  expected 


value  for  each  cell  is  computed  using  the  following  formula; 


Using  the  formula  above,  we  arrive  at  the  following  expected  frequencies  for  each  cell 


outlined  in  Table  4. 


Table  4.  Expected  Cell  Frequencies. 


hiscal  Year 

Protest 

NO  h^rotest 

. 

94 

22.616 

20.484 

95 

32.465 

29.535 

96 

43.462 

39.538 

97 

2/.229 

2T77T 

Period 

Protest 

N6"Prote8r 

Pre-FASA 

41.014 

Post-FASA 

64.014 

7UW 

The  top  cells  include  the  expected  cell  frequencies  for  the  first  analysis  between 
each  of  the  years  while  the  lower  half  contains  those  for  the  pre-  post-FASA  comparison. 
According  to  the  results,  the  data  meets  the  assumption  that  the  expected  frequency  for 
each  cell  should  be  at  least  five.  The  next  assumption  requires  that  the  scale  of 
measurement  be  at  least  nominal.  The  data  exceeds  this  requirement  since  the  scale  of 
measurement  is  ratio  and  the  measurements  enable  the  determination  of  how  many  times 
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as  much  of  the  measured  characteristic  is  possessed  by  one  unit  of  the  sample  than 
another. 

The  next  assumption  requires  that  the  samples  are  independent.  Samples  are 
independent  if  the  assumption  that  “b”  has  occurred  does  not  alter  the  probability  that  “a” 
occurs  [McClave  and  Benson,  1994: 160].  In  this  instance,  the  occurrence  of  one  protest 
does  not  necessarily  alter  the  probability  that  another  award  will  be  protested. 
Additionally,  an  award  that  was  protested  one  year,  does  not  necessarily  change  the 
probability  that  other  awards  will  be  protested  in  any  of  the  following  years. 

The  final  assumption  requires  that  the  data  in  each  sample  is  frequency  in  form 
and  categorized  into  mutually  exclusive  and  exhaustive  categories.  Since  the  data 
consists  of  the  number  of  protests  as  a  percentage  of  the  number  of  contract  awards  and 
either  the  contract  is  protested  or  not  protested,  this  assumption  appears  to  have  been  met 
as  well. 

Given  that  the  assumptions  have  been  met,  it  appears  that  the  Chi-square  test  will 
be  the  most  appropriate  for  this  data  analysis.  In  order  to  use  this  test,  the  data  is 
compiled  and  formatted  into  table  form.  The  following  tables  depict  the  form  that  will  be 
used  for  both  analyses. 


Table  5.  Contingency  Table  For  First  Analysis. 


Fiscal  Year 

Protest 

NO  Protest 

4  otals 

93 

30 

h 

35 

94 

24 

15 

43 

95 

43 

19 

62 

55 

38 

45 

- 55 

97 

9 

43 

52 

1  otals 

T4T 

TTT 

275 
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Table  6.  Contingency  Table  For  Second  Analysis. 


No  Proiesi 

'24 

'47 

GO 

00 

vrz 

Table  5  depicts  the  data  for  the  first  analysis  which  is  intended  to  discern  whether 
there  is  any  difference  between  the  individual  years.  Table  6  depicts  the  data  for  the 
second  analysis  which  is  intended  to  test  the  hypothesis  that  there  has  been  a  change  in 
the  frequency  in  protests  between  the  pre-  and  post-FASA  periods.  The  hypotheses  being 
tested  by  the  first  analysis,  along  with  the  tests  statistic  and  rejection  region  are  as 
follows: 


Ho:  Pi=P2=P3=P4=  Ps 
H,:  Pl^P2^P3^P4^  Ps 


Test  statistic: 


Rejection  region:  Reject  Ho  if  exceeds  9.348  (from  a  Chi- 
square  distribution  table)  based  on  a  significance  of  .05  and  (r-l)(c- 
1)=4  degrees  of  freedom. 

The  results  of  this  analysis  revealed  that  the  X^of  50.05  exceeded  9.348,  therefore 
we  reject  the  null  hypothesis  in  this  analysis.  By  rejecting  the  null  hypothesis,  we  can 
conclude  that  at  least  one  of  the  years’  protests  differs  from  the  others’.  This  conclusion 
does  not  answer  the  question  whether  there  are  differences  between  the  pre-  and  post- 
FASA  periods,  however.  Since  there  is  an  apparent  difference  in  the  protests,  we  now 
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split  the  data  into  two  categories  consisting  of  pre-  and  post-FASA  to  test  for  impacts  of 
the  Act  on  protest  frequency.  The  second  analysis  is  very  similar  to  the  first,  except  we 
will  be  looking  at  whether  the  two  period,  pre-FASA  (fiscal  years  93-94)  and  post-FASA 
(fiscal  years  96-97)  are  different.  The  hypotheses  being  tested  by  the  second  analysis, 
along  with  the  tests  statistic  and  rejection  region  are  as  follows: 

Ho:  Px  =  P2 
Ha:  Px  ^  P2 

Test  statistic:  = 

ij 

Rejection  region:  Reject  Ho  if  exceeds  3.841  (from  a  Chi- 
square  distribution  table)  based  on  a  significance  of  .05  and  (r-l)(c- 
1)=1  degrees  of  freedom. 

The  results  of  this  analysis  revealed  that  the  of  23.48  exceeded  3.841,  therefore 
we  reject  the  null  hypothesis  in  this  analysis  as  well.  Rejecting  the  null  hypothesis  in  this 
instance  indicates  that  there  is  a  statistically  significant  difference  in  the  protest  frequency 
between  the  pre-  and  post-FASA  periods. 

Summary 

In  the  analysis  of  this  data,  there  were  some  surprising  outcomes.  Some  of  the 
responses  from  the  interview  questions  exceeded  the  decision  criteria,  indicating  that 
certain  areas  of  source  selection  practices  have  changed.  These  results  are  somewhat 
contradictory  to  what  was  implied  from  the  research  conducted  during  this  study’s 
literature  review.  Additionally,  the  statistical  analysis  of  the  archival  data  revealed  that 
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there  has  been  a  change  in  protest  frequency,  which  is  also  contradictory  to  the 
insinuations  of  the  literature  review  findings.  The  next  chapter  discusses  the  findings  of 
the  analyses  in  this  chapter  and  their  implications  on  the  acquisition  function  of  the  Air 
Force. 
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V.  Conclusions  and  Recommendations 


General 

In  the  conclusion,  a  brief  recapitulation  of  the  findings  detailed  in  chapter  IV  will 
be  presented;  also  conclusions  will  be  drawn  as  to  the  import  of  those  findings.  Finally, 
brief  suggestions  will  be  made  as  to  areas  for  further  research  that  were  suggested  by  the 
findings  of  this  thesis. 

Restatement  of  Objectives 

This  research  was  intended  to  provide  insight  as  to  the  effectiveness  of 
Acquisition  Reform,  specifically  the  Federal  Acquisition  Streamlining  Act  of  1994,  at 
reducing  the  frequency  of  bid  protests.  By  analyzing  the  effectiveness  of  the  Act,  the 
goal  of  this  study  was  to  discern  whether  FASA  has  actually  changed  contracting 
practices  and  if  so,  what  impact  these  changes  may  have  had  on  protest  frequency.  As 
previously  discussed  in  chapter  II,  evidence  uncovered  during  the  literature  review 
seemed  to  indicate  that  FASA  had  negligible  affect  on  actual  contracting  practices  and 
consequently  has  neither  increased  nor  decreased  protest  frequency. 

Impact  on  Contracting  Practices 

As  indicated  in  chapter  III,  a  case  study  methodology  employing  structured 
telephone  interviews  was  used  in  determining  whether  any  changes  in  contracting 
practices  has  occurred  as  a  result  of  FASA.  The  results  and  conclusions  of  those 
interviews  follow  and  are  summarized  in  Table  7. 
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Table  7.  Decision  Matrix  For  Interview  Responses.  One  or  more  of  any  question  marked 
with  an  asterisk  that  receives  >=  80%  of  the  responses  will  indicate  that  FAS  A  has 
changed  actual  contracting  practices  within  AFMC. 


Question 

0) 

CO 

c 

o 

Q. 

CO 

a> 

q: 

CONCLLBION 

Decision  | 

■ 

1 

I 

ESC  % 

1  %  OHS 

Support  1 

o 

CL 

CO 

■ 

1 

1 

S3 

100 

H 

100 

m 

100 

A 

FASAdid  not  result  intiepractioe  of  prowiding  an 
increased  amxrt  of  informstionto  uBucoessful 
ofeors  during  post-aftsrd  ctoiefe 

1 

7 

0 

20 

0 

0 

0 

3a 

A 

RASA  resulted  in  the  fwactice  of  prcwicfing  an  imoeased 
annouit  of  inforrration  to  unsuccessful  offgas  diring 
post-award  debriefs  (ranking) 

1 

33 

60 

20 

20 

33 

33 

■ 

1 

60 

40 

80 

60 

56 

67 

■ 

A 

FASA  charged  the  resporse  time  to  reque^  by 
UTBUco^sful  offerors  for  post-aA@nd  debriefs 

•k 

67 

60 

60 

78 

SO 

4a 

rag 

■ 

33 

20 

40 

40 

22 

50 

5a,6a 

■ 

FASA  char^  essliBtion  f^a  practices 

H 

13-7 

0 

204) 

20 

11 

17 

B 

FASA  has  not  changed  equation  factor  fxactioes 

7,0 

0 

400 

0 

11 

0 

Insuffidert  evidmoe  easts  to  moke  determination 

80-93 

100 

20-100 

K) 

83 

A 

FASA  changed  e^^lustion  toor  practi^ 

* 

67-60 

80 

40-20 

B 

80 

83 

HH 

B 

FASAdid  rx3t  change  ewsiL^tion  fada 

■ 

33-33 

20 

20 

20 

TT 

m 

IrsL^ciat  ewidenoe  e&dsfe  to  make  d^emiinatton 

■ 

mm 

mm 

m 

mm 

■il 

■ 
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FASA  changed  contracti  ng  practices 

It 

93 

100 

El 

100 

89 
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B 

FASAhas  not  changed  contracting  practices 

7 

mi 

im: 

0 

11 

■SI 

Tr~ 

_ 

0 

0 

0 

0 

0 

-11 

Questions  1  and  2  dealt  with  changing  the  content  of  post-award  debriefs.  As 
indicated  by  Table  7,  the  responses  invariably  indicated  that  organizations  throughout  the 
command  have  increased  the  amount  of  information  provided  to  unsuccessful  offerors 
during  post-award  debriefs  as  a  result  of  FASA’s  implementation.  The  one  individual 
who  did  not  agree  maintained  that  his  organization  had  always  provided  adequate 
information  during  debriefs  and  that  the  content  of  these  debriefs  met  or  exceeded  the 
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level  called  for  in  the  provisions  of  FAS  A.  Of  all  the  questions,  these  two  were  among 
the  ones  that  had  the  highest  degree  of  agreement  across  all  categories. 

The  responses  received  from  question  3  were  quite  different  however.  Question 
3  dealt  with  providing  ranking  information  during  post-award  debriefs  and  the  responses 
received  were  not  overly  conclusive  in  either  direction.  Most  of  the  interviewees  seemed 
to  indicate  that  not  a  lot  of  change  has  occurred  in  this  area,  and  based  on  the  decision 
matrix  above,  it  seems  that  FASA  did  impact  this  arena  of  the  contracting  process. 

A  slight  majority  of  the  respondents  indicated  that  their  organizations  have 
expedited  responses  to  requests  for  debriefs  to  be  in  concurrence  with  the  Act.  However, 
according  to  the  decision  criteria  in  the  matrix,  only  one  sub-command  indicated  with  a 
sufficient  proportion  that  practices  have  actually  changed.  This  is  not  necessarily  bad 
news  though  since  those  that  disagreed  that  any  change  has  occurred  unanimously 
maintained  that  their  existing  policies  met  or  surpassed  the  response  timing  called  for  in 
the  Act. 

Questions  5  and  6  focused  on  changes  in  evaluation  factor  practices.  The  results 
indicate  with  sufficient  proportion  across  both  demographic  groups  and  all  sub¬ 
commands  with  the  exception  of  one  that  there  has  been  a  change  in  this  area.  The  one 
sub-command  that  maintained  that  no  change  has  occurred  however,  disagreed  in  such  a 
large  proportion  with  the  other  groups  as  to  drop  the  overall  AFMC  level  below  the 
required  80%  level.  Consequently,  we  cannot  conclude  that  FASA  changed  evaluation 
factor  practices  for  the  purposes  of  this  study.  The  subjects  in  ESC  which  disagreed  with 
the  others  on  this  area  indicated  that  the  provisions  in  FASA  pertaining  to  use  of  past 
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performance  as  evaluation  factors  and  identification  of  significant  evaluation  factors  and 
their  relative  importance  in  solicitations  was  common  practice  within  their  organizations 
prior  to  the  Act’s  implementation.  They  also  went  on  to  say  that  FASA  did  not  alter  their 
practices  in  this  area  as  well. 

The  last  set  of  questions  in  the  decision  matrix  included  an  open-ended  question 
to  gain  the  interviewees  insight’s  into  the  impact  of  FASA.  The  respondents  indicated  in 
sufficient  proportions  that  FASA  has  changed  contracting  practices  in  their  organizations. 
The  magnitude  of  the  postulated  changes  varied  across  the  respondents  from  relatively 
small  to  consequential,  with  a  majority  agreeing  the  most  significant  area  of  change  to  be 
in  debrief  content. 

An  additional  set  of  questions  not  included  in  the  matrix  was  intended  to  gain 
PCO’s  insights  into  any  quality  changes  in  offerors’  proposals  since  FASA.  Evidence 
uncovered  during  the  literature  review  suggested  that  an  important  and  significant  impact 
of  FASA  would  occur  in  the  area  of  debrief  content.  This  led  the  author  to  hypothesize 
that  future  proposals  could  potentially  improve  if  unsuccessful  offerors  are  truly  getting 
more  information  regarding  their  weaknesses  in  competitions.  The  results  of  this  query 
were  not  conclusive  as  a  slight  majority  felt  that  there  had  been  no  change  in  proposal 
quality  with  the  implementation  of  FASA.  Responses  to  this  question  were  used  to 
address  proposition  #3  and  the  results  indicate  that  FASA  has  not  impacted  post-award 
protest  frequency  within  the  United  States  Air  Force.  Interestingly,  one  respondent 
felt  that  proposal  quality  has  degraded  as  a  result  of  the  Act  because  contractors  have 
become  overly  reliant  on  the  “openness”  and  “generosity”  of  the  Government  in 
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providing  information  to  offerors  and  early  industry  involvement.  As  a  caveat  to  this 
implication  that  contractors  are  becoming  lazy,  the  author  would  like  to  add  that  this  was 
not  the  impression  given  by  the  other  respondents. 

Based  on  the  findings  of  these  interviews,  the  author  concludes  that,  in  answer  to 
Proposition  #1,  FASA  impacted  Air  Force  source  selection  procedures  as  they  relate 
to  content  and  administration  of  post-award  debriefs.  This  impact  amounts  to  an 
increase  in  the  content  provided  to  imsuccessful  offerors  during  post-award  debriefs. 

More  specifically,  unsuccessful  offerors  appear  to  be  receiving  more  information 
regarding  reasons  for  their  exclusion  and  weaknesses  in  the  proposals  they  submitted. 
Additionally,  contracting  procedures  overall  have  been  impacted  by  FASA.  While  there 
is  some  question  as  to  the  magnitude  of  these  changes,  it  is  apparent  that  procedures  have 
changed  nonetheless  and  these  changes  could  potentially  influence  the  frequency  of 
protests. 

While  it  appears  that  an  answer  to  the  first  proposition  has  been  found,  there  are 
some  interesting  additional  observations  in  the  responses  when  they  are  broken  down 
demographically.  Referring  to  Figure  12,  there  appears  to  be  somewhat  of  a  disparity  in 
the  way  ESC  responded  in  comparison  to  the  other  two  commands,  particularly  on  the 
critical  questions  marked  with  an  asterisk.  The  respondents  from  ESC  seemed  to  agree 
with  the  other  two  commands  on  all  of  the  critical  questions  with  the  exception  of  one, 
but  to  a  lessor  degree.  Those  that  disagreed  with  the  others  indicated  that  the  mandates  of 
FASA  were  already  being  practiced  prior  to  the  Act’s  implementation.  They  seemed  to 
suggest  that  their  organization  was  on  the  leading  edge  of  acquisition  reform  and  was 
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proactive  in  improving  their  procedures.  This  phenomena  was  emphasized  by  the 
responses  to  questions  5c  and  6c.  When  broken  down  demographically,  all  groups  with 
the  exception  of  ESC,  felt  with  sufficient  proportion  of  responses  (80%)  that  there  has 
been  changes  in  evaluation  factor  practices  at  their  organizations  as  a  result  of  FASA. 
ESC  on  the  other  hand  was  in  strong  disagreement  on  this  issue,  with  agreement 
receiving  only  40%  and  20%  on  each  of  the  questions.  Once  again,  their  reasoning  for 
disagreement  was  that  their  organization  was  more  progressive.  Given  this,  it  is 
interesting  to  note  that  most  of  those  at  ESC  that  did  feel  that  FASA  had  an  impact  on 
ESC’s  contracting  procedures  felt  that  the  biggest  change  occurred  in  providing  more 
thorough  debriefs.  This  would  seem  to  contradict  the  opinions  of  others  in  that  same 
group.  Another  interesting  outcome  was  that  although  ESC  appeared  to  have  the  largest 
contingent  with  the  opinion  that  nothing  had  changed,  they  also  had  the  largest  contingent 
with  the  opinion  that  protests  have  increased.  Adding  to  this  contradiction  was  the  fact 
that  none  of  the  ESC  respondents  felt  that  there  has  been  no  change  in  protest  frequency 
as  opposed  to  the  other  commands  which  did  have  a  few  with  that  opinion.  In  other 
words,  some  at  ESC  appear  to  be  asserting  that  procedures  have  not  changed  since  their 
organizations  are  progressive,  but  protests  have  changed.  It  would  be  interesting  to  learn 
why  they  feel  there  has  been  a  change  in  protests  despite  the  progressive  contracting 
methods  they  employ. 

The  only  large  disparity  in  responses  when  broken  down  by  function  appeared  to 
be  in  question  7c.  A  majority  of  support  personnel  felt  that  protests  have  decreased, 
while  their  counterparts  in  the  SPOs  felt  that  there  has  been  no  change.  They  were 
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somewhat  in  agreement  that  procedures  have  changed  as  a  result  of  FAS  A,  however  they 
did  not  see  eye-to-eye  on  the  impact  on  protests.  It  was  encouraging  to  note  however  that 
the  number  of  respondents  in  both  groups  that  felt  that  protests  have  increased  was 
relatively  small  and  comparable. 

The  question  as  to  whether  there  has  been  any  change  in  protest  frequency  was 
addressed  through  statistical  analysis  of  archival  data  and  is  answered  in  the  following 
section. 

Protest  Frequency 

The  Chi-square  tests  conducted  in  chapter  IV  resulted  in  the  rejection  of  the  null 
hypothesis  in  both  cases.  As  a  result,  there  is  sufficient  evidence  to  suggest  that  there  has 
been  a  difference  in  protest  frequency  between  the  pre-  and  post-FASA  periods.  The 
analysis  conducted  between  each  of  the  years  confirmed  that  the  samples  are  not  drawn 
from  identical  populations.  Although  there  was  a  slight  spike  in  protest  frequency  in 
1996,  the  number  of  protests  as  of  June  30*  of  fiscal  year  1997  are  quite  low  as  compared 
to  all  previous  years  and  overall,  protests  have  decreased  between  the  pre-  and  post- 
FASA  periods.  The  next  section  summarizes  the  results  and  derive  conclusions  as  to 
their  importance. 

Conclusions 

The  findings  of  the  analyses  have  already  been  presented  however  in  order  to 
draw  any  conclusions  it  is  necessary  to  refer  back  to  the  design  of  the  study.  This  study 
was  designed  as  a  two  phase  study,  with  the  overall  objective  of  determining  whether 
FASA  has  impacted  award  protest  frequency  within  the  United  Stated  Air  Force.  The 
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first  phase  consisted  of  structured  interviews  to  determine  what,  if  any,  changes  in 
contracting  source  selection  practices  have  occurred  as  a  result  of  FAS  A.  The  results  of 
this  phase  indicated  that  practices  have  changed  within  AFMC,  particularly  in  the  area  of 
post-award  debriefs  to  unsuccessful  offers.  As  a  result  of  these  findings  we  are  able  to 
reject  the  hypothesis  formed  by  proposition  #1  that  FASA  has  not  impacted  the  United 
States  Air  Force’s  source  selection  processes  in  a  manner  that  would  significantly 
affect  post-award  protest  frequency.  These  findings  by  themselves  are  quite  interesting 
however  further  analyses  had  to  be  conducted  in  the  second  phase  to  determine  whether 
these  changes  resulted  in  changes  in  the  frequency  of  award  protests. 

The  second  phase  was  intended  to  analyze  protest  frequency  within  ASC  prior  to, 
and  after  FASA’s  implementation.  The  results  of  this  analyses  indicated  that  protest 
frequency  has  changed  between  the  two  periods  with  what  appears  to  be  an  overall 
reduction  in  frequency.  Figure  1 1  delineates  how,  with  these  findings,  we  can  draw 
conclusions  as  to  their  meaning. 

The  shaded  areas  indicate  the  course  the  study  has  taken  as  a  result  of  the 
findings.  Since  it  was  determined  that  FASA  has  changed  contracting  practices  and  there 
has  been  an  overall  reduction  in  protests,  we  are  able  to  reject  the  hypothesis  formed  by 
proposition  #2  that  FASA  has  not  impacted  post-award  protest  frequency  within  the 
United  States  Air  Force. 
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Figure  11.  Study  Design. 


While  the  data  collected  was  limited  to  AFMC  for  the  surveys  and  ASC  for  the 
protest  numbers,  the  findings  should  be  representative  of  the  Air  Force  as  a  whole  since 
AFMC  manages  approximately  52%  of  the  annual  United  States  Air  Force  budget  and 
represents  the  largest  buying  organization  in  that  service.  Additionally,  all  agencies 
within  that  branch  are  governed  by  the  Federal  Acquisition  Regulation  (FAR). 
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Implications 

As  a  result  of  the  methods  employed  in  this  research,  it  appears  that  FASA 
resulted  in  a  decrease  in  the  frequency  of  award  protests.  What  does  this  mean?  It  means 
that  Government  efforts  at  reducing  protests  appear  to  be  working.  The  degree  of  their 
successfulness  is  questionable  however. 

As  indicated  from  the  findings  of  the  interviews  and  literature  review,  there 
appears  to  be  some  end-users  in  the  field  with  unpopular  opinions  of  acquisition  reform 
and  their  affects  in  general.  This  warrants  the  question  as  to  whether  their  claims  have 
merit  or  are  they  merely  resisting  change.  After  all,  many  of  the  acquisition  reform 
efforts  prescribe  paradigm  shifts  from  structured  procedures  and  centralized  decision¬ 
making  to  decentralized  decision-making  and  employing  the  principles  of  creativity  and 
empowerment  in  problem  solving.  Those  with  a  bureaucratic  mindset  might  be  locked 
into  one  management  style  and  be  unwilling  to  submit  to  such  principles.  It  is  possible, 
however  that  those  with  negative  attitudes  towards  such  initiatives  might  have  sharper 
insight  into  the  impacts  md  those  in  majority  are  caught  up  in  a  euphoria  that  change 
equates  to  progress  and  improvement.  If  on  the  other  hand,  the  opposite  is  true  and  the 
“nay-sayers”  are  merely  resisting  change,  could  their  attitudes  be  contributing  to  a 
negative  inertia  towards  reform  and  improvement?  If  this  were  the  case,  it  is  conceivable 
that  acquisition  reform  could  be  more  successful  in  their  goals  of  streamlining 
acquisitions,  improving  competition,  and  reducing  protests  if  these  opinions  could  be 
swayed  or  reduced  through  training  or  education.  This  would  be  particularly  true  of 
efforts  such  as  FASA  which  appear  to  be  working  despite  some  negative  influence. 
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Despite  whether  the  negative  opinions  have  merit,  they  do  exist  and  must  not  be 
ignored,  for  their  impact  could  be  notable.  This  leads  to  the  question  of  what  could  be 
contributing  to  some  of  the  negative  attitudes  towards  acquisition  reform?  It  is  possible 
that  it  is  just  a  mind-set  as  previously  mentioned.  On  the  other  hand,  perhaps  some  are 
risk  averse  and  do  not  feel  comfortable  with  the  reduction  in  controls  and  set  procedures, 
and  increased  “openness”  called  for  by  FASA.  Those  that  view  acquisition  reform 
unfavorably  may  perceive  there  to  be  “work-arounds”  to  those  procedures  that  they 
disagree  with  or  feel  uncomfortable  abiding  by,  which  negates  any  necessity  to  take 
acquisition  reform  seriously. 

Some  may  assert  that  reform  is  used  as  a  “hollow”  effort  by  politicians  to  grab 
headlines.  When  this  occurs,  everyone  jumps  on  the  bandwagon  and  embraces  the  new 
buzzwords  and  concepts  to  appear  progressive  and  forward-thinking.  According  to  those 
with  these  opinions,  once  the  novelty  of  the  effort  fades,  its  back  to  business  as  usual  and 
everyone  forgets  all  about  it.  Another  related  perception  is  that  the  time  lag  in 
implementing  acquisition  reform  efforts  is  excessive  and  is  caused  by  “red-tape.”  In  the 
words  of  another  anonymous  contracting  officer,  “we  hear  a  lot  of  hooray,  but  don’t  see 
the  parade.”  Once  the  changes  reach  the  end-user  level,  they  are  watered-down  and  fall 
short  of  their  expectations.  This  was  echoed  in  the  Literature  Review,  were  it  was  learned 
that  FASA  fell  short  of  expectations  and  could  have  went  a  lot  further  in  the  eyes  of  a 
few.  If  this  is  true,  perhaps  the  Government  needs  to  concentrate  on  streamlining  its 
policy  implementation  as  well  as  its  procedures. 
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Another  opinion  encountered  was  the  lack  of  user  involvement  in  formulating 
new  acquisition  efforts.  An  example  was  given  by  one  contracting  officer,  who  wished  to 
remain  anonymous,  was  the  Air  Force’s  acquisition  reform  efforts  known  as  the 
“Lightening  Bolts.”  One  of  the  provisions  of  this  initiative  was  the  creation  of  a  SPO 
manpower  model  which  established  limitations  on  the  number  of  personnel  within  those 
organizations.  According  to  the  unnamed  contracting  officer,  the  numbers  seemed  very 
arbitrary  and  the  creators  (SAF/AQ)  should  have  gotten  inputs  from  those  effected  in 
order  to  better  understand  the  implications.  It  makes  perfect  sense  to  involve  the  end- 
users  in  the  creation  of  new  procedures  and  policies  and  if  it  is  not  being  done,  the 
effectiveness  of  these  initiatives  could  definitely  be  hampered.  If  end-user  involvement  is 
being  practiced,  it  might  be  beneficial  to  examine  why  there  is  still  a  perception  it  is  not. 

In  addition  to  the  finding  that  FAS  A  changed  contracting  procedures,  this  study 
also  revealed  that  protest  frequency  has  also  decreased.  While  it  is  inappropriate  to  imply 
a  causal  relationship  exists  between  the  implementation  of  FAS  A  and  the  reduction  in 
protest  frequency,  the  findings  of  this  study  do  suggest  that  there  is  a  degree  of 
correlation  between  the  two.  It  is  possible  that  other  factors  contributed  to  the  decline  in 
protests  as  well,  however. 

An  attempt  was  made  to  minimize  the  influence  of  other  factors  on  protest 
frequency  in  this  study  however  complete  isolation  was  impossible  for  obvious  reasons. 
The  impact  of  reductions  in  Govenunent  procurements  and  the  changes  in  the  types  of 
procurements  and  systems  being  acquired  could  still  provide  varying  degree  of  influence. 
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Among  the  defense  firms,  there  is  a  trend  that  could  diminish  the  impact  of  not 
winning  a  Government  source  selection.  As  defense  spending  shrinks,  an  increasing 
number  of  firms  that  relied  solely  on  DOD  contracts  are  turning  to  the  civilian 
marketplace.  Some  of  these  firms  either  get  out  of  the  defense  business  altogether  or 
supplement  that  sector  with  commercial  products  directed  at  other  markets.  This  trend 
essentially  reduces  the  impact  of  not  winning  a  Government  contract  since  these 
companies  are  no  longer  completely  reliant  on  that  market.  Similarly,  the  recent  trend 
towards  mergers  and  teaming  arrangements  among  the  large  defense  firms  such  as 
Boeing-McDoimell  Douglas,  Lockheed-Martin  and  Northrup-Grumman  could  reduce  the 
impact  of  not  winning  a  Government  contract  as  well.  This  shrinking  effect  on  the 
defense  industrial  base  could  conceivably  make  it  more  likely  that  a  company  could 
simultaneously  be  a  prime  contractor  involved  in  a  competition  and  have  subsidiaries  as 
subcontractors  with  other  prime  contractors  bidding  on  the  same  contract.  Even  if  the 
company  is  not  awarded  the  contract  as  the  prime  contractor,  the  impact  of  losing  is 
diminished  because  they  are  still  getting  a  piece  of  the  action  through  one  of  their 
subsidiaries.  As  the  impact  of  losing  a  competition  diminishes,  it  is  conceivable  that  the 
attractiveness  of  protesting  would  diminish  as  well. 

Another  interesting  notion  is  that  perhaps  the  appeal  of  protesting  has  diminished 
as  a  result  of  some  of  the  changes  in  Government  policies,  such  as  the  use  of  contractor 
past  performance  as  evaluation  factors.  Contractors  may  misunderstand  the  ramifications 
of  what  type  of  information  is  considered  under  past  performance  and  believe  that 
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protesting  could  give  the  impression  that  they  are  difficult  to  work  with  and  harm  their 
chances  of  winning  future  contracts. 

As  previously  mentioned,  the  findings  of  this  study  should  be  acceptable  for 
generalization  to  the  Air  Force  as  a  whole.  However,  insights  gained  through  the  course 
of  this  study  put  the  degree  of  generalization  in  suspect.  While  FASA  appeared  to  have 
made  notable  changes  in  contracting  practices  overall,  perhaps  the  most  significant 
impact  occurred  in  operational  contracting  at  the  individual  unit  level.  The  provisions  of 
FASA  made  significant  modifications  to  small  procurement  procedures  by  establishing 
preferences  for  commercial  off  the  shelf  items  and  the  use  of  “micro-purchase” 
procedures  for  procurements  less  than  $2500.  The  scope  of  this  study  was  limited  to  the 
systems  arena  of  Air  Force  contracting  and  based  on  the  findings,  the  author  believes  that 
the  findings  in  a  similar  study  of  operational  contracting  would  yield  the  same 
conclusions  with  greater  impact. 

It  feels  good  to  say  that  things  are  getting  better,  however  it  is  difficult  to  do  so 
with  absolute  certainty.  While  the  conclusions  of  this  study  are  somewhat  profound,  new 
questions  arise  and  it  becomes  necessary  to  address  them  if  we  desire  to  advance  our 
knowledge  of  such  actions  and  their  impacts.  Additionally,  there  are  limitations  to  the 
findings  of  this  study,  which  will  be  addressed  next. 

Limitations 

This  research  was  somewhat  bounded  by  the  element  of  time  and  availability  of 
data.  The  findings  of  this  research  might  have  been  more  robust  had  protest  data  prior  to 
1 993  been  available.  This  would  have  helped  to  isolate  the  affects  of  previous  acquisition 
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reform  efforts  from  the  impacts  of  FAS  A  and  minimize  the  effects  of  extreme 
fluctuations  in  protest  frequency.  Agencies  are  not  required  to  track  this  data  by  contract 
amount  and  the  researcher  was  very  fortunate  that  one  agency  in  ASC  did  so  anyway. 

The  time  limitation  also  applies  because  the  Act  has  been  in  affect  for  a  relatively 
short  time.  It  would  be  interesting  to  conduct  a  follow-on  study  to  determine  if  the 
affects  of  FAS  A  change  after  it  has  been  in  effect  for  a  longer  period.  As  time  passes,  it 
becomes  more  admissible  that  the  end-users  of  the  legislation  become  more  proficient 
with  the  mandates  and  procedures  of  the  Act,  or  perhaps  some  latent  flaws  or  defects  will 
surface  as  a  function  of  experience  with  the  provisions.  To  adequately  ascertain  the 
impact  of  the  Act,  these  issues  will  have  to  be  addressed  as  they  coalesce. 

One  other  limitation  was  the  possible  impact  other  events  or  acquisition  reform 
efforts  may  have  had  on  protest  frequency.  It  would  be  nearly  impossible  to  completely 
isolate  these  effects,  however  the  findings  from  this  research  seem  strong  enough  to 
indicate  that  FASA  may  have  reduced  protests  to  some  degree.  Conducting  the  same 
type  of  analysis  in  the  future  would  be  convoluted  by  the  implementation  of  FARA, 
which  occurred  shortly  after  FASA  and  was  considered  to  be  a  refinement  of  FASA’s 
provisions.  Additionally,  it  is  difficult  to  attribute  the  reductions  in  protests  to 
contractors  being  more  satisfied  with  procedures.  There  are  other  factors  that  could 
potentially  influence  such  decisions  to  protest,  such  as  risk  analysis,  or  the  down-sizing  of 
the  defense  industrial  base.  These  issues  warrant  further  investigation  as  well  to  isolate 
the  impacts  of  acquisition  reform. 
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Recommendations  For  Further  Study 


During  the  course  of  this  study,  several  new  questions  were  raised.  Some 
recommendations  for  further  study  are  as  follows: 

•  Conduct  similar  interviews  with  industry  to  gain  their  perceptions  of  the 
effectiveness  of  FASA  and  acquisition  reform  in  general  at  making 
Government  procurements  more  appealing.  The  findings  of  this  study 
suggests  that  protests  have  decreased  since  FASA’s  implementation.  Further 
research  is  necessary  to  discern  the  impact  of  other  possible  external  factors 
which  could  influence  the  frequency  of  award  protests.  It  is  possible  that 
factors  other  than  FASA  could  have  influenced  this  reduction  in  protests. 

•  Conduct  a  similar  study  to  discern  the  impacts  of  FARA  on  contracting 
practices  and  protest  frequency.  The  findings  of  this  study  indicate  that  FASA 
did  impact  contracting  practices,  particularly  in  the  source  selection  area,  and 
it  appears  that  protests  may  have  decreased  as  a  result.  It  would  be  interesting 
to  study  whether  FARA  had  a  similar  impact. 

•  Conduct  further  research  on  the  differences  that  were  encountered  in 
responses  between  sub-commands.  On  particular  questions  on  the  interview, 
there  was  considerable  disparity  in  responses  between  the  three  commands.  In 
many  of  those  instances,  the  respondents  indicated  that  the  reason  for  the 
disparity  was  that  their  organizations  were  perhaps  more  progressive  and 
implemented  procedures  before  they  were  mandated  by  FASA.  Further 
investigation  of  these  responses  could  lead  to  a  better  understanding  of  how 
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procedures  differ  between  the  different  commands  and  how  these  changes 
could  impact  protests. 

•  Conduct  a  similar  study  using  other  DOD  or  executive  branch  agencies.  Some 
of  the  findings  from  the  Literature  Review  indicated  that  FASA’s  impact 
might  have  been  even  more  significant  on  some  of  the  other  agencies  that  had 
less  structured  procurement  procedures  than  that  of  the  Air  Force.  It  would  be 
interesting  to  understand  FASA’s  impact  on  protest  frequency  in  such 
organizations. 

•  Conduct  a  study  of  the  “negative”  attitudes  towards  acquisition  reform  in  an 
attempt  to  discover  there  foundations  and  merit.  It  appears  that  there  is  an 
element  that  views  such  reform  unfavorably  and  could  impede  the 
effectiveness  of  those  efforts.  Developing  a  better  understanding  of  such 
attitudes  could  perhaps  make  future  acquisition  reform  efforts  more  effective. 

•  Conduct  a  study  to  discern  the  benefits  and  drawbacks  of  current  tracking 
metrics  to  measure  the  effectiveness  of  acquisition  reform  efforts  and 
suggestions  for  improvements  or  additions.  As  previously  mentioned,  some 
of  the  data  that  was  used  in  the  archival  analysis  was  not  required  to  be 
maintained,  but  was  very  useful  in  ascertaining  FASA’s  effectiveness. 

Summary 

The  direct  and  indirect  costs  of  protests  can  be  substantial  to  the  Government  as 
well  as  industry.  The  Government  recognizes  the  negative  impact  protests  have  on  the 
procurement  process  and  has  enacted  several  laws  and  regulations  in  an  effort  to  reduce 
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protest  frequency  and  streamline  procurement  procedures.  In  conducting  this  study,  the 
author  sought  to  determine  whether  one  particular  initiative,  the  Federal  Acquisition 
Streamlining  Act  of  1994,  was  effective  in  accomplishing  that  objective. 

The  findings  of  this  study  suggest  that  FASA  did  have  an  impact  on  decreasing 
protests.  The  degree  of  impact  is  questionable,  however  and  there  are  external  factors 
that  could  have  potentially  influenced  the  protest  frequency.  Additionally,  evidence 
uncovered  during  the  Literature  Review  and  structured  interviews  suggests  that  perhaps 
the  Act  could  have  been  more  effective  if  negative  connotations  towards  acquisition 
reform  in  general  could  be  reduced. 
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Appendix  A;  Structured  Interview  Script 


1)  Pursuant  to  FAS  A,  has  your  organization  increased  the  amount  of 
information  provided  to  lonsuccessful  offerors  during  post-award 
debriefs  regarding  reasons  why  they  were  excluded  from  further 
consideration? 

a)  Yes  -  go  to  question  2 

b)  No  -  go  to  question  la 

c)  Don’t  know  -  go  to  question  2 

la)  Which  best  describes  why  you  think  that  the  enactment  of  FAS  A  has 
not  resulted  in  an  increase  in  the  amount  of  information  provided  to 
unsuccessful  offerors  during  post-award  debriefs  regarding  reasons 
why  they  were  excluded  from  further  consideration? 

a)  This  practice  was  already  being  used  in  my  organization 
prior  to  FAS  A.  -  go  to  question  2 

b)  My  organization  has  not  yet  implemented  a  policy  or 
regulation  pertaining  to  such  provisions.  -  go  to  question  lb 

c)  I  don’t  know.  -  go  to  question  lb 

d)  Other.  -  go  to  question  lb 

lb)  Do  you  expect  your  organization  to  implement  FAS  A  provisions  into 
its  post-award  debrief  procedures  regarding  the  inclusion  of  reasons 
why  unsuccessful  offerors  were  excluded  from  further  consideration? 

a)  Yes  -  go  to  question  2 

b)  No  -  ask  why,  then  proceed  to  question  2 

c)  Don’t  know  -  go  to  question  2 

2)  Pursuant  to  FASA,  has  your  organization  increased  the  amount  of 
information  provided  to  unsuccessful  offerors  during  post-award 
debriefs  regarding  weaknesses  in  their  proposals? 

a)  Yes  -  go  to  question  3 

b)  No  -  go  to  question  2a 

c)  Don’t  know  -  go  to  question  3 


89 


2a)  Which  best  describes  why  you  think  that  the  enactment  of  FAS  A 
has  not  resulted  in  an  increase  in  the  amount  of  information 
provided  to  imsuccessful  offerors  during  post-award  debriefs 
regarding  weaknesses  in  their  proposals? 

a)  My  organization  provided  this  information  on  a  level  that 
met  or  surpassed  that  called  for  by  FASA  prior  to  its 
implementation.  -  go  to  question  3 

b)  My  organization  has  not  yet  implemented  a  policy  or 
regulation  pertaining  to  such  provisions.  -  go  to  question  2b 

c)  I  don’t  know.  -  go  to  question  2b 

d)  Other  -  go  to  question  2b 

2b)  Do  you  expect  your  organization  to  implement  FASA  provisions  into 
its  procedures  to  provide  unsuccessful  offerors  with  information 
about  weaknesses  in  their  proposals  during  post-award  debriefs? 

a)  Yes  -  go  to  question  3 

b)  No  -  ask  why,  then  proceed  to  question  3 

c)  Don’t  know  -  go  to  question  3 

3)  Did  your  organization  include  the  overall  ranking  of  all  offerors 
during  post-award  debriefs  to  unsuccessful  offerors  prior  to  FASA? 

a)  Yes  -  go  to  question  4 

b)  No  -  go  to  question  3a 

c)  Don’t  know  -  go  to  question  3a 

3  a)  Does  your  organization  now  include  the  overall  ranking  of  all  offerors 
during  post-award  debriefs  to  unsuccessful  offerors? 

a)  Yes  -  go  to  question  4 

b)  No  -  go  to  question  3b 

c)  Don’t  know  -  go  to  question  4 

3  b)  Which  best  describes  why  you  think  that  the  enactment  of  FASA  has 
not  resulted  in  your  organization  providing  the  overall  ranking  of 
all  offerors  to  unsuccessful  offerors  during  post-award  debriefs? 

a)  That  provision  is  not  applicable  to  my  organization. 

-  go  to  question  3c 

b)  My  organization  has  not  yet  implemented  a  policy  or 
regulation  pertaining  to  such  provisions.  -  go  to  question  3c 

c)  Other.  -  go  to  question  3c 

d)  Don’t  know  -  go  to  question  3c 
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3c)  Do  you  expect  your  organization  to  implement  FASA  provisions 
into  its  procedures  and  begin  providing  the  overall  ranking  of  all 
offerors  during  post-award  debriefs? 

a)  Yes  -  go  to  question  4 

b)  No  -  ask  why,  then  proceed  to  question  4 

c)  Don’t  know  -  go  to  question  4 

4)  Pursuant  to  FASA,  has  your  organization  changed  its  response  time 
to  debrief  requests  made  by  unsuccessful  offerors? 

a)  Yes  -  go  to  question  5 

b)  No  -  go  to  question  4a 

c)  Don’t  know  -  go  to  question  4a 

4a)  Which  best  describes  why  you  think  that  the  enactment  of  FASA 
has  not  resulted  in  a  change  in  your  organization’s  response  time 
to  debrief  requests  made  by  unsuccessful  offerors? 

a)  My  organization’s  response  time  to  debrief  requests  already 
met  or  surpassed  the  response  timing  called  for  by  FASA. 

-  go  to  question  5 

b)  My  organization  has  not  yet  implemented  a  policy  or 
regulation  pertaining  to  such  provisions.  -  go  to  question  4b 

c)  I  am  unfamiliar  with  the  provisions  of  FASA  pertaining  to 
response  time  to  requests  for  post-award  debriefs.  -  go  to 
question  5 

d)  I  am  unfamiliar  with  my  organization’s  policy  on  response 
time  to  requests  for  debriefs.  -  go  to  question  5 

e)  Other  -  go  to  question  4b 

4b)  Do  you  expect  your  organization  to  implement  FASA  provisions 
into  its  procedures  and  change  its  response  time  to  debrief  requests 
made  by  unsuccessful  offerors? 

a)  Yes  -  go  to  question  5 

b)  No  -  ask  why,  then  proceed  to  question  5 

c)  Don’t  know  -  go  to  question  5 

5)  Did  your  organization  include  contractor  past  performance  as  an 
evaluation  factor  prior  to  the  enactment  of  FASA? 

a)  Yes  -  go  to  question  5c 

b)  No  -  go  to  question  5a 

c)  Don’t  know  -  go  to  question  5a 
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5a)  Does  your  organization  now  include  contractor  past  performance 
as  an  evaluation  factor? 

a)  Yes  -  go  to  question  6 

b)  No  -  go  to  question  5b 

c)  Don’t  know  -  go  to  question  5b 

5b)  Do  you  expect  your  organization  to  implement  FASA  provisions 
into  its  procedures  and  start  using  contractor  past  performance  as 
an  evaluation  factor? 

a)  Yes  -  go  to  question  6 

b)  No  -  ask  why,  then  proceed  to  question  6 

c)  Don’t  know  -  go  to  question  6 

5c)  Did  the  implementation  of  FASA  change  the  way  your 

organization  used  contractor  past  performance  as  an  evaluation 
factor? 

a)  Yes  -  go  to  question  6 

b)  No  -  go  to  question  6 

c)  Don’t  know  -  go  to  question  6 

6)  Did  your  organization  identify  in  its  solicitations,  all  significant 
evaluation  factors  and  their  relative  importance  prior  to  the 
enactment  of  FASA? 

a)  Yes  -  go  to  question  6c 

b)  No  -  go  to  question  6a 

c)  Don’t  know  -  go  to  question  6a 

6a)  Does  your  organization  now  identify  in  its  solicitations,  all 
significant  evaluation  factors  and  their  relative  importance? 

a)  Yes  -  go  to  question  7 

b)  No  -  go  to  question  6b 

c)  Don’t  know  -  go  to  question  6b 

6b)  Do  you  expect  your  organization  to  implement  FASA  provisions 
into  its  procedures  and  start  identifying  in  its  solicitations,  all 
significant  evaluation  factors  and  their  relative  importance? 

a)  Yes  -  go  to  question  7 

b)  No  -  ask  why,  then  proceed  to  question  7 

c)  Don’t  know  -  go  to  question  7 
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6c)  Did  the  implementation  of  FAS  A  change  the  way  your 
organization  identified  in  its  solicitations,  all  significant 
evaluation  factors  and  their  relative  importance? 

a)  Yes  -  go  to  question  7 

b)  No  -  go  to  question  7 

c)  Don’t  know  -  go  to  question  7 

7)  Overall,  would  you  say  that  FASA  changed  your  organization’s 
contracting  process? 

a)  Yes  -  go  to  question  7a 

b)  No  -  go  to  question  8 

c)  Don’t  know  -  go  to  question  8 

7a)  What  would  you  say  is  the  most  significant  change?  -  go  to 
question  7c 

7b)  Why  don’t  you  believe  that  FASA  significantly  changed  your 
organization’s  contracting  process?  -  go  to  question  8 

7c)  Overall,  would  you  say  that  award  protests  have  increased  or 
decreased  as  a  result  of  these  changes  made  in  your  organizations 
contracting  process? 

a)  Increased  -  go  to  question  7d 

b)  Decreased  -  go  to  question  7e 

c)  Neither  -  go  to  question  8 

d)  Don’t  Know  -  go  to  question  8 

7d)  What  single  change  do  you  feel  had  the  greatest  impact  on 
increasing  the  frequency  of  award  protests? 

a)  Increase  in  information  provided  to  unsuccessful  offerors 
during  post-award  debriefs.  -  go  to  question  8 

b)  Shortened  response  time  to  unsuccessful  offerors  requests 
for  debriefs.  -  go  to  question  8 

c)  Use  of  past  performance  as  an  evaluation  factor.  -  go  to 
question  8 

d)  Other  -  go  to  question  8 

e)  Don’t  know  -  go  to  question  8 
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7e)  What  single  change  do  you  feel  had  the  greatest  impact  on 
decreasing  the  frequency  of  award  protests? 

a)  Increase  in  information  provided  to  unsuccessful  offerors 
during  post-award  debriefs.  -  go  to  question  8 

b)  Use  of  past  performance  as  an  evaluation  factor.  -  go  to 
question  8 

c)  Other  -  go  to  question  8 

d)  Don’t  know  -  go  to  question  8 

8)  Do  you  feel  that  the  quality  of  proposals  submitted  have  changed  in 
regards  to  completeness  and  responsiveness  to  the  solicitation  since  the 
implementation  of  FAS  A? 

a)  Improved  -  go  to  question  8a 

b)  Degraded  -  go  to  question  8b 

c)  No  change  -  survey  complete 
c)  Don’t  know  -  survey  complete 

8a)  What  do  you  think  contributed  the  most  to  these  improvements  in 
proposals  as  a  result  of  FAS  A? 

a)  Increase  in  information  provided  to  unsuccessful  offerors 
during  post-award  debriefs.  -  survey  complete 

b)  Use  of  past  performance  as  an  evaluation  factor.  -  survey 
complete 

c)  Other  -  survey  complete 

d)  Don’t  know  -  survey  complete 

8b)  What  do  you  think  contributed  the  most  to  the  degrading  in 
proposals  as  a  result  of  FASA? 

a)  Increase  in  information  provided  to  unsuccessful  offerors 
during  post-award  debriefs.  -  survey  complete 

b)  Use  of  past  performance  as  an  evaluation  factor.  -  survey 
complete 

c)  Other  -  survey  complete 

d)  Don’t  know  -  survey  complete 

e)  Other  -  survey  complete 
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Appendix  B:  Structured  Interview  Responses 


Table  8.  Overall  Responses  Within  AFMC. 
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Table  9.  Responses  Within  ASC. 


96 


Table  10.  Responses  Within  ESC. 


a 

b 

c 

d 

e 

5T — . 

4  /  80% 

0 

0 

0 

Qla 

1  /  100 

0 

0 

0 

0 

SIE 

0 

0 

0 

0 

0 

Q2 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

br— 

0 

5  /  100 

0 

0 

0 

1  /  20% 

0 

0 

0 

|Q3B 

0 

0 

0 

0 

0 

ggggg 

0 

0 

0 

gim^g[ 

2  /  40% 

0 

0 

0 

2  /  100 

0 

0 

0 

0 

|[Q?E 

0 

0 

0 

0 

0 

M _ 

ausa 

0 

0 

0 

gygggl 

ggggg 

0 

0 

0 

Q56 

0 

i— Ilia 

0 

0 

0 

q5c 

0 

0 

0 

Q6 

0 

0 

0 

0 

0 

0 

0 

0 

0 

p66  1 

0 

0 

0 

0 

0 

gani 

ggggg 

0 

0 

1 

igigg 

BBBa 

0 

0 

0 

0 

gQigig 

2  /  50% 

0 

0 

1 

0 

0 

0 

0 

0 

2  /  100 

0 

0 

i  0 

i  0 

|Q^  1 

^gUgg 

0 

4  /  80% 

0 

0 

0 

0 

ggggi 

0 

0 

PBB  1 

0 

0 

0 

0 

0 

97 


Table  11.  Responses  Within  SMC. 
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Table  12.  Responses  By  Type 
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